Family dynamics are very hard to break. In many cases, they will do anything to avoid what they consider to be embarassment....that indeed is their issue, not yours.
I did not speak to my mother for 10 years, I was just plain tired of the drama and abuse.
What happended? Well, we have been back together for 7 years, she understand my bounderies, and, she knows I will enforce them, for me. Yes, she is still drinking, however, all the crap is over, done.
So, are you selfish? No, you need to do what you need to do for you, you cannot change him or them.
Take care of you, let the chips fall where there may.