I am so sorry Tombplant, stuff like this is so very painful. It will ease with time, and with you forging your own way forward without this bloke whom I don't happen to like. Sorry, but...
I think the final paragraph in your post sums it all Tomb, "why can't I let him go when...."
You can't let him go because he fulfills some need in you. Some co-dependent need if I may be so bold. I've been there too, so don't be angry with me for saying.
We have to forge our own independent paths Tomb, even when in a relationship. I'd be focussing on
myself at the moment. Not him.
He's asked you to leave, and that's what you must do. Do it with grace; and eventually with all the good work you put into
yourself you'll find yourself streets ahead of this cruel little man.
You mightn't know it right now, but you really
don't need him. You need you.