Unfortunately, without a restraining order and/or his willingness to leave, it's unlikely you can "make" him leave. I had to make it clear to my AH (and follow through) that I would take our girls and I would be the one to leave if he didn't. Thankfully since his image matters to him, he realized he'd look like a bigger a$$ than he already is if he forced his kids to live elsewhere bc he wouldn't leave, so he left and the girls and I are here.
I think that if you want to be away from him, as inconvenient/unfair (and it is, I agree) as it is, you are going to have to be the one to leave if he won't. Is there anywhere you can go for a while?