Yep, I agree with Zoe on you being seen as a threat, the enemy.
I felt so horrible back in the earlier years with my AL when his dirty 'friends' (drinking buddies) were constantly belittling me, backstab, saying rude things blahblahblah. I was genuinely hurt -I thought these people were my friends. I also felt as if I had to defend and justify myself to him and to them about what his people had said etc. The art of detachment in it's highest form worked well for me in this situation. I'm sure it still goes on to some degree, but they can no longer get a reaction out of me or my attention. If AL wants to hang with them, that is his agenda. I no longer stick my head in the hornets nest by hanging out socially with them. On occasion I have to briefly encounter them and I am always cordial and polite, smile and move on. It took me a long time to realize this -years-, but taking the high road worked best for me.
I believe that ALL men (AL or not) will usually have the one, icky best friend that you don't much care for. BOUNDARIES