Like Lexie said, he is abiding by your boundary. So, what's your REAL boundary?
You know now what he thinks. And that hurts, I've been there. And you know what you think. So now you have to decide whether ever the twain shall meet. I'm fairly certain that waiting for other people to change to fit in with me is going to be a looooong wait, so I have to change either my thinking or my situation.
Which one is it going to be? Because, as it stands you are in stalemate.