When my XAH feels like getting nasty with me, he tells me I'm controlling and crazy just like my mother, which he knows pushes all of my primal buttons. But guess who between us has actual unchecked mental health issues and all the controlling quirks that go with it?
He's probing your boundaries to make sure he still has an effect on you. What he thinks about you -- what this degenerate, cheating, active alcoholic thinks about you -- should have no bearing on your self-worth. He might take a look in the mirror instead of projecting his negative feelings on you.