i can understand the fear you have, you never know how far a person will go when they are an addict let alone behaving in the way that he has. I would call the police BUT as that would probably cause problems...i would look for another place to live and then when witnessing his abusiveness call the police.
I would suggest finding a place to live seperate from the relationship that is unhealthy going on here between your mother and him.
i know you said you do not want to leave your 60 year old mother alone with him and i can understand that but there isnt much you can do being there either in my opinion. hes going to do what he is going to do and shes going to allow it to happen. perhapse you should speak with your mother when your brother is not present and ask her if she would go with you to an alanon meeting. this would be healthy for the both of you and possibly help her to see what she is truly dealing with here. she is not dealing with just "her son" nor is she dealing with someone who is "like her" as she probably feels like her having a personailty disorder puts her in the same boat as him. he needs help and your mother is trying to be doctor. it never works.