After I got sober, I found being around heavy drinkers wasn't fun. They seemed immature and shallow. But when I was drunk, they seemed brilliant and funny. Then probably like you, the next night I would call one of them from a bar to come join me, but would end up being told they had something else they had to do? Like you, I was the one left drinking after everyone else had quit. I was feeling like the outcast again. This was also true on a larger scale in my life. My old college buddies were doing grown up things most of the time, and I was still getting drunk. It was for me anyway. It was for me anyway.
You feel like an outcast when you aren't drinking, and you feel like an outcast when you are. If you are an alcoholic, your default state is being drunk, so feeling like the outcast when you are sober is your excuse to drink. On the other hand, feeling like an outcast when you are drunk has no desirable solution, because it would involve not drinking to fit in, and that is way out of your comfort zone.
The bottom line is that your bigger issue is that you are an alcoholic, not that you are an outcast. In fact, I would argue that the whole outcast issue is just a bunch of baloney you use to get drunk.