You didn't force him to go there - he could have gone anywhere. If he wanted to be in a healthier environment, he would have found one.
I'm sure you and the kids miss him - he's been a part of your lives....however, you didn't send him away because he was healthy, or a great dad or husband. It's easy to convince ourselves of the "if's" the "if onlys" and of the "maybe one days."
She might be a trigger for him, sure....but he could leave if he wanted to.