Well, you've told him if he left again that you are DONE. He left. So are you done? If so, you could change the locks. Tell him to get some boxes and come get his stuff, arrange a time when your brother can be there with you and give him a 2 hour window to pack up. He's obviously got other options of where he can lay his head at night.