Your recovery is just as important as your AH. I too would be greatful if my AH asked me to hear him speak. But only because it would not be something out of the ordinary for him and because I would love to hear what he had to say.
I understand your are worried about what he would say. But the old saying goes, there's nothing to fear but fear itself. Facing those fears makes you a better person.
You could go with him to show him your support and if his speech is going somewhere you don't want to go, there is no law that says you have to stay in the room. Would he support you if you were the one giving the speech?