This is the hard part of early recovery, but it's essential in order to start to heal. I'm glad you know you shouldn't go. I think you don't need to give anyone a reason if you don't want to. I found it incredibly liberating to know that I could 'No, thanks' when it was something I didn't want to do. You can give them an explanation if you want to, but you don't owe them an explanation.