You are not going to far. You are being a great mom.
My experience.
When I was in your position I did not go far enough. I let them go with him when I should not have. I let him in my house when I should not have. I went the 'extra mile' when I should not have. My xah also eventually moved away. My boys were/are heartbroken that they do not have a father in their lives. They miss him. It hurts. But they are better and happier now with him physically gone then with the unease and stress they had when he was close by and drinking. I didn't fully get it at the time. I get it now that I can look back at the big picture in hindsight. I know it is hard to hear the quacking and see the look in the little one's eyes but stand strong. It makes me sad and ashamed that I didn't create stronger boundaries on my children's behalf.
Stick to your instincts. You are doing the right thing.