surething,
Once you have self-identified yourself as a non-drinker, you have nothing to explain. It's just who you are. You aren't really "not" doing anything, you are just making choices a non-drinker makes.
There are a lot of other things you are not doing at the same time -- do you need to justify to the world everything you choose not to do?
If someone is unhappy with your non-participation in an activity you no longer choose to be a part of, leave. That is not selfish. You are doing them a favor if they are that uncomfortable with you.