I concur that people who have not been through severe, prolonged depression just are incapable of understanding what it does to a person, and I can relate to some of your experiences with others. This is similar to trying to comprehend what it is that addiction does to people when you simply don't know. Put them together ... but let's not go there today.
I am fortunate that medication and therapy has done a lot for me in dealing with my depression and anxiety disorder. How well the maladies in this area are controlled or are relieved differs much between individuals. I can not drink and somebody may not really notice, but if I have that "face" people do get the body language immediately. When my brain function is fogged by anxiety (or by medication), people react to that, too. I react to those things, and when I do it's usually some form of withdrawal from the situation around me, even in the past to the point of dissociation. I tend to protect myself as much as possible from being stigmatized by only addressing my experience in such matters to my closest people, and after a serious conversation with my family, they know when not to push my buttons by suggesting things that aren't going to make me feel better in that moment.
Some people, I just ignore what they say about addiction, depression and other mental disorders.
I hope that in society at large we will see less stigma around people who are going through addiction recovery or who are trying daily to manage their mental function. The popular media can help with that, but sometimes they just further the notion that we are selfish and maladapted to living in the normal world. I wish we didn't have to be part of a secret world to such an extent.