Its simply not an easy process for anyone.
I think you have done what you could - you answered the phone and you told him the truth. You aren't leading him on and that's the kindest thing you can do for him.
And that's what you need to continue to do if you want to make this easy. When you get tired of the phone calls or you start feeling your own self weaken its time to block them.
You did good - really proud of you its hard to hurt someone, even when they earned it.