My situation and profession forces me to be at a lot of "happy hours" or places where drinks were being served.
At first I feared it, but then I found that every time I said, "no thanks, I don't drink," it strengthened my identity as a sober person. Hearing myself say it out loud and sometimes answer the "why?" question helped ingrain it into my head that I am just someone who doesn't drink.
I compare it to not eating meat. I have been a vegetarian since I was 15. If I see someone eating a hamburger (I like hamburgers) I might think, that smells good, or that probably tastes good, but the thought never enters my mind that "I might eat one of those... no one would find out" (as it does with alcohol). Being a vegetarian is a strongly integrated part of who I am. Eating meat is just outside the realm of things I might do.
I'm hoping that being a sober person eventually occupies a similar position in my thoughts and self-conception. Being around people who are drinking, and telling them that I don't drink, helps reinforce that.
That being said -- I think it's entirely normal to feel uncomfortable when everyone is drinking. It sounds like you are doing the right thing by only putting yourself in those situations when necessary.
GG