I seriously, seriously doubt that he is that malicious. Is he trying to wound you a little? Probably. He's hurt so he's hurting back. People do that. Sick people are more likely to do it without thinking through whether it's fair, or how it is really affecting someone else.
And he's also trying to turn it around, to change the story in his head to soothe himself. It isn't right, it isn't fair, but the very best thing you can do is what you've already done. Don't give him a forum to say these things to you.
Hugs, I'm sorry you are feeling so awful. But just remember, it doesn't mean that much. It's a sick person, flailing.