Nah, doesn't bother me a bit. I just politely decline. I don't owe anyone an explanation. Nobody ever put a gun to my head and forced me to drink. I think you'll find that most people are wrapped up with themselves and in the long run, don't really care if you drink or not.