By and large I've found it to be true - with the exception of my drinking buddies and my family...and both of those groups drink pretty heavily and knew me as a mad drinker.
Most people don't really give two hoots if I'm drinking alcohol or not - seriously.
Those who don't like it get used to it, or they gravitate away.
D