Bodhi, I went years without getting together with family for that very reason. Drinking, boundary-crossing, poking "fun", control issues all around, questioning my every move. Literally.
You can always go for an hour or two and then leave. That would be my compromise to my husband, but I sure wouldn't spend a lot of time with them if they're drinking.