atalose has it right. As heartbreaking as it can be, you have to set up boundaries for yourself.
I used to get involved in triangulation drama between my sister and my parents, and all it did was stress me out to the point that _I_ started drinking every night. Fortunately, my body does not respond well to alcohol, and what is a pleasurable experience for so many people was not a pleasurable experience for me. I also started grinding my teeth to the point that I broke a filling. I finally went to therapy last year and it helped me loads.
Your mom seems to be in a domestic violence situation. I would encourage her to contact her the DV hotline to at least talk to somebody in a confidential manner. Depending on how old your mom is there are organizations such as
https://elderabuse.org.uk/ that might be of some assistance to you.
My cousin was terribly abusive to the point that his mom put a deadbolt on her bedroom door because she was afraid he was going to kill her. He sexually abused my sister. She covered up everything to the point that his younger siblings ended up in foster care. I'm not telling this to scare you - I'm telling you this because you may not even conceive how far your mom will go to protect your brother. And you can't take it personally, and you can't take it upon yourself to force her to act. The only thing you can do is point her in the right direction.
I had to do this with my own parents. I know they're terribly disappointed that I don't have much of a relationship with my sister at all. But just as I don't try to force them to stop their enabling, they no longer try to force a relationship between us.
I do feel anger towards my sister, but I also know that she chooses to live the life she has because my cousin inflicted so much pain on her she'd rather pretend it never happened at all. She bullied me too when we were younger but now I can view it through the lens of pity rather than the lens of anger.
I wish you strength and peace for the journey ahead.