I worried about the same thing. It ended up ugly and I kicked XAH out of my house. My plan before that had been to let him know to start looking, and one week before he actually moved out to go stay elsewhere for a week or two, both myself and my children. I also had removed anything I feared he would take and I could not live without per the advise of my attorney. Cleaned out what was rightfully mine from our bank accounts, etc.
I hope you continue to work on you!