emp, may I suggest, to protect yourself and the children, that you at least do up "legal separation" papers, giving you legal custody of the kids? I say that because it kinda sounds like he's gearing up for something. I mean, is he also being manipulative and playing mind games with you, but it could lead to something more. And if you could set up visitation at a neutral location, that would probably help out too.
I'm a little sensitive about this stuff right now, a local news story is about a murder/suicide, the man was an alcoholic who was supposed to be in a halfway house a few hours away, wife did have a protection order out against him.