Where is your brother the rest of the time? With your/his mother?
It sounds like he should be there ALL the time.
Your brother is actually in danger here. That puts everything else in second place in the list of priorities. Can you talk to his/your mother, or whoever he is with when he's not with your dad? They may not know how bad the situation is, but they need to.
No child should need to live like that. (((hugs)))