My fiancee likes to drink beer occasionally. I have a real hangup about beer....because of prior bad experience (that is putting it mildly). I told him that I couldn't deal with having beer in the frig. Fine, it is not that important to him. Now, every once in awhile he stops at the c-store and buys only one. I can deal with that.
For years, I had escape routes from gatherings where the drinking bothered me. And I still use them, but no longer get invited to those much. It just became evident that I was uncomfortable and would isolate myself.