There were times when I had to treat my AH as if he was just a friend and than I had to decide if I would let my kids do something/be with this friend.
It was those times when I stopped letting AH come by whenever he felt like it and it was those times when I had to cancel my weekend plans because I wasn't comfortable with having DS spend the weekend with his AD.
It was a time I was so overwhelmed (not as much as you can get with younger kids, but enough) that I was a basket case about AH not being there for the kids and a friend of mine looked at me and said - you are still trying to rely on your AH and the truth is you can't. It was then that I stopped taking responsibility for my AH relationship with my kids and just did what was best for them and for me.
That was over a year ago and I am just now starting to see the recovery in myself and saw from the beginning that my kids were just fine.
Reach out to others as much as you can for support. Only you can make the choices, but healthy people can and do provide support for where we are in life.