This guy needs to go. Kudos for you believing your gut.
Also, kudos for waiting first, to let your anger cool off, and verify that you're making a sound decision, and not just reacting.
It is a sound decision. Fire this self-serving moron.
In spite of everything going on emotionally, wanttobehealthy, it sounds like logically you are in a very good place, and I am glad.
Judging by his responses on the phone, too, he didn't sound like he was interested in protecting your interests, anyway.
You don't need this guy.
Good for you!