Wow! You surely did get alot of feedback.
I believe I'd be saying something like "I am happy for you and your recovery but I am just not ready to go hear you speak."
It took me forever to understand that I was entitled to my feelings they are neither right nor wrong.
When I keep the conversation in I statements I am better understood by the other person. Just as I can't force anyone else to do something, I don't want to feel force either. I dig in my heels and won't budge, if I am left alone to decide "when and if" I don't feel pressured.
Another suggestion is to have someone tape his talk before hand so you can listen to it in private before you decide if you want to go.
I always fear the unknown and it sounds to me like you may be as afraid of your re-actions as to what you will hear him say.
Good luck either way. Sending happy thoughts your way.