(officially anyways!)
I'll say what I always do: talk to a few lawyers. Get informed and get organized. My XAH is also quite lazy and he quickly dropped a bunch of visitation because "the daycare is too far" or "I'm not feeling well (read: hung over", or "todays is a bad day". It's now been 9 months since he's seen our daughter. I assume that he'll eventually disappear.
It sounds like you already know what you want and that it'll be better for you and your children. AWESOME!