In my opinion it just seems like you need to tell your boyfriend how much this upsets you. He might think that he is protecting you by keeping you out of the picture with her. If you are honest with him and things stay the same, well then really all you can do is decide whether the relationship is worth staying in or not in the long term.