First of all, I wanted to welcome you to the site. Besides this fabulous board, we also have a whole forum dedicated to Friends and Family (of alcoholics and/or addicts). Maybe you could check them out and work on getting some help for yourself! There are wonderful and supportive people all over this place!!
IMHO, forcing an alcoholic into treatment will probably backfire more often than not. In order to really stop, the alcoholic has to want to stop. However, allowing your brother to continue staying at your mother's house no matter what he does is not doing him ANY good. Maybe you and your mom could sit down with him and let him know that if things don't change SOON, he will have to find another place to live. I know it's tough--especially for a mom--but it could be the best thing for him in the long run.