I had that same feeling. There were no physical scars that people could see but there are emotional and mental scars. My XABF used to say, "Well, everybody drinks" and if you look at the society now - it is kind of true. Binge drinking happens so much and the weekend seems to be about getting as drunk as possible.
Really though, you don't have to validate your reasons for leaving him. You were unhappy and some people leave because of that alone! But the things he said were reason enough too. You should never have to justify yourself to anyone, and if I were you, I wouldn't. Nowadays, I tell people that my XABF had an alcohol and drug problem and that's why I left. And I have never been questioned further about it.