I agree, I wouldn't go.
I do understand that you feel obligated to your husband, but surely you can come up with a healthy way to hang out with him. How about going out for brunch this weekend or to a movie? If you feel about these people the way you do, you shouldn't go and your husband should accept that.