If he's going to start a fight because your mom is there, can you tell him to leave now? given that you know he's going to be pissy anyway, get it all over and done with? tell him your mom is coming to babysit the kids and you want him out now, that way you have some support and a witness/someone to keep the kids occupied whilst you tell him. Tell him he's got 10mins to get his stuff together and then you'll call the police? I know you are exhausted, but imagine the peace once he's gone: he's blackmailing you and holding you hostage.
this is pathetic and despicable behaviour on his part, you are not the bad guy for throwing him out, can you imagine doing the same in reverse? squatting in your ex's house against their wishes, using the kids to blackmail them? he is the bad guy here.