Part of me thinks I should just run now! She is only 5, and the baby is 1.. Its like gambling, I could run for the hills now, and play it safe so the kids won't remember much.
I think the bigger gamble, would be to wait around and hope that things get better. That he matures. That he doesn't relapse.
Ask yourself, what is the WISE decision here, based on his past and present behavior.
Don't worry about him. He is an adult and he can take care of himself. If he wants to see his kids he'll make it work. But it's your job to protect your babies. He's not going to. Obviously. He's more concerned about "being intimate". My ex was the same way. He has a lot of issues to work out before he can be a truly capable parent.