Why don't you move out when your mom does? You can say it's to "give him space" or whatever. But I'd also spring it on him at the last minute. I mean, if you're going to have movers there, or friends, or whoever will be helping your mom move, might be worth it to jump then.
The thought of being along with him is scaring me, and I don't even know him!