That's hard. I am so sorry you're dealing with this. Handing them over to him must break your heart.
I had similar issues with AXH. He would text me and tell me I needed to deal with some discipline problem when the kids were with him. Because I was worried about how he would handle it if I didn't, I would do that at first. But after a while I refused. I'm not criticizing you - my kids were older and capable of getting the hell out of the house if necessary.
I ended up telling him that "if you want to be a parent, you will have to find constructive ways of interacting with the children." He didn't, and it all blew up, and he started avoiding having the kids over because it was too stressful for him... which, in the end, was better for the kids. It was hell while it lasted, though. They would call me at all odd hours crying and telling me he was being mean, and all I could say was "since you have to be there, you have to find a way to handle it."
Their way of handling it ended with him giving up custody.