Yuck indeed.
I completely get this. I often feel empathy for my XAH even though he is also the person I cannot stand the most in this world. I think that when you share children with someone, there is always some sort of feeling there, want it or not. You want them to thrive and be a good parent, to be well.
This is not a bad thing. You don't want to lose empathy in life. You are taking the right actions, and that is what is important.