No, it's not how he felt about you, it's how he felt about himself.
Projection at its finest. You were his mirror. You reflected back to him what he was/is.
I never listen to people with bad words about me. I know it's how they feel about themselves, because I live in a way where I really try not to hurt people but still be within my boundaries.
But people who dislike themselves so much feel sooo much better when making other people feel just as shetty.
In my experience, when people are telling me "what I am", I just listen and let them rant (as today with a friend), because I know it's all the things they wish they could tell themselves. And I just love.
Now, if it's a regular thing where he's treating you like crap and saying these things, then just calmly state that he can no longer talk to you that way. It's unacceptable. And walk away. When he starts, you walk away. I had to do that with an ex (older daughter's father)...yea. Took a while, but he finally "got it" that his opinion of me wasn't any of my business.