Hello Lunchbox,
It certainly sounds like a complicated dynamic. You are the only one in the middle of it and know your family best. You do care, and your decisions are based around that fact.
Whether or not any of you go on this vacation is entirely up to you. If you go, and he goes as well, then I hope you and the children can find a way to enjoy yourselves in spite of your husband's ill humor about the whole thing.
I hope today will be a peaceful one!