I know exactly what you mean. That's where the focusing on yourself comes in.
You know that if you speak to him you will, at the most, agree to a conversation and maybe even meeting up. At the very least, you have a conversation that leaves you hurt.
Neither of those things are good things for you.
So it's not about caring for him or not, it's clear that you do and there is nothing wrong with that. The goal is not to hate him or even dislike him. The goal is to protect yourself. If talking to him means hurting yourself and you are willing to do that, then who is that helping? Not you, probably not him?
And if it is helping him, at what cost to you?
Could you talk to him now without being hurt?