If he were to be trusted with the children alone .... then it seems as if you would know that. He obviously sets off a mom's pre programmed protection alarm. My ex was abusive in every way. When I had him removed from our home, he had supervised visitation at the house of a neutral party for two hours once a week. He was angry and felt humiliated. In denial about his potential for physical violence when drinking. So his non drinking self on those nights - had to suffer these feelings.
Too bad. I was afraid that he would take our youngest daughter or be drunk etc. This sufficed until he stopped coming and all regular visitation ended. The court did not force visitation. We were spared that battle.
It's a tough time and I wish you all the strength that you need. I never regretted leaving. We were apart for 13 years after the divorce and he died this past spring. He wasted his mind and his life. He had so little to do with his daughters. One ended up as an angry adult who makes little effort to stand on her own feet. The other is a gem. He didn't drink in the home but he would come home after being out. It's a bad cycle - if you can spare your children this type of exposure - then please do it. It's irreparable. I'm proud that you are conscientious and protective. This makes a huge difference in their young lives and as they grow into the world. Best to you, hugs, Joie