If that's the truth of what he felt, that's the truth of what he felt.
I think to put it behind us, on either or both sides, we have to be totally honest. Otherwise, the wound festers, even when it is covered over.
If what you did hurt him that bad, then it hurt him that bad. If he is telling you this to punish you, that is one thing, and it isn't recovery.
If he is telling you this because that is how he truly felt, then you need to (eventually) own that your behavior had that consequence for someone else.
Shakespeare said "True compassion is ruthless".
ShootingStar1