A good way to explain your position is to file for legal separation or divorce. I suggest you contact an attorney to get the ball rolling. Even if you're not ready for a divorce, legal separation will set rules for who can stay in the house (if his name is on the deed/lease he can come and go at will unless there is an order that YOU have exclusive possession for now), child support, visitation, etc. Without an order there would be nothing to prevent him from picking up the kids at school and keeping them with him.
Get your ducks in a row. Serve him with papers and he will get the message.