I don't remember the back story but is he abusive?
If not, is there a way to set clear boundaries/sleeping arrangements in the house until you decide what you need to do?
Not sure of the ages of your kids either but this is a lot of drama for them (pulling them out of the home abruptly) so the more careful you can do this, the better for all.
I guess what I'm saying is as long as you feel your kids and your SAFETY is not at risk around him, then look into other living arrangements that won't cost you the job, etc etc.
There is a differecne between 'acting' and 'reacting'.
Stop and catch your breath.