From what you write, you and I were similar drinkers. When I tell people I don't drink, I just say that the older I get, the more I felt sick from alcohol, so I decided to cut it out and now I feel better. No one can argue with that. I have had one or two ask why I don't just have one or two and I just answer, "for me it was easier to just avoid it all together."
Really, no one but a complete ass will push it farther than that.