I like a definition I heard about detachment, to paraphrase.
"We set a boundary. What we detach from is the other person's reaction to our having set that boundary. "
Anger is a common reaction on the other person's part. Also the phrasing your friend used is very common ("what's the big deal," telling you he's not going because of your response). Labeling you as "unfair, cruel, disrespectful" is also common. I swear, there is a web site somewhere where people go to get these phrases because they are used universally!!
Hang in there. Find the poem "My Life in 5 Chapters." Each chapter begins, "Im walking down the street and I see a hole." I used to see it circulating at Alanon meetings. You might find yourself in one of the chapters.