Yep, I second Jen's post. Don't wait for him to be happy for you to leave. XAH was mad that his rent and truck bill payor, one-who-kept-the-cable-on left. Even after he moved in with a new enabler GF, he was astounded and POd when I filed for divorce.
Take care of yourself and your kids. It's more than OK to do that.