Ignore the insults your brother makes towards you, put your son in daycare or find another sitter, and stand firm when your sister says you are overreacting towards your son being around your brother. You can't please everyone, you can only please yourself. You are not pleased with the current situation so change it.