Guilt has taken you hostage today because we allow ourselves to be controlled by negative or toxic emotions. Guilt is directly related to people pleasing, we codies want everyone to like us, to accept us.
Personally, I do not care what others think of me. I know that I am doing the very best that I can, if that is not good enough for others....tough dingleberries.
If they are so concerned about your ex, they can adopt him and he can go live with them. Bet their tune would change very quickly!
Sending hugs your way!