Hi,
Yes, it's hard. And, it can take a lot of changes in early recovery, in order for it to work. I had to get rid of a few toxic people from my life, including a family member. I had to accept things about myself that I didn't want to accept, in order to be able to begin recovery. But, you can do this.
I couldn't be around people drinking alcohol for a long time after I stopped drinking, and I still don't do it very often. I don't want to.