I have 2 children with anaphylaxis to peanuts and tree nuts. They both get about 5 birthday party invites a year. Each time, I remind or let the parents know, make sure they can handle an epi pen. Many times the cakes are not safe for my kids to eat. They never miss a party, they just don't eat the cake. They love Halloween and have awesome costumes, I have to take over half their candy away. (I take candy from children lol)
It will take time to deal with the anxiety. I hope it happens soon for you.