Thank you so much for your responses, they really mean a lot!
I have been seeing a therapist and have actually been diagnosed with PTSD because of the crazyness of the week before he went to rehab. Also, I'm setting up some therapy for my kids who I'm sure have been affected as well. He left in a dramatic fashion.
After reading your posts and reflecting I think my anxiety was coming from a combination of things. First and foremost the PTSD, secondly from the change in my behaviour by actually speaking up, third because, like goatgirl said, I know he's going to have to answer to what came up during that call (which is a good but scary thing because jeez when was the last time he had to take responsibility for his actions) and fourth...was he going to be mad about that.
I'm so thankful that I've reached out for help through therapy and through SR. Your guys' experience and perspectives in similar situations was super helpful. My view of my relationship with him changes daily and looks nothing like it did even a few months ago.
Thanks again!