Yes, I can imagine. People do what they want to do, whether it's selfish and inconsiderate or not. His family has a right to guzzle booze and your H has a right to remove himself from the situation.
I don't drink. I don't go to bars. It's not a matter of refraining from drinking to accommodate my AH. He's gonna drink whether I do or not. And he does. Almost constantly. I have nothing to do with it. It is no longer my concern. If I had a burning desire to have a mixed drink or a glass of wine, I suppose I would.
However, I made the choice for myself, regardless of what AH choses to do, that I would not drink.
If you want to enjoy a drink with friends, by all means do. It would probably be better not to do it with your H around. But then, you know him and you know whether or not it would bother him.