Liz,
So what if "I just don't want to go" won't cut the mustard with him? Has his recent behavior cut the mustard with you? You do not have to defend your decision to him or anyone else. Do what's right for you and your son.
My son tells me that people who are on the fence about a decision should flip a coin. If you feel a sinking feeling in your stomach at the results of the flip, you already had your mind made up. Heads you go, tails you don't. If heads turn up and you are bummed out about that, you really don't want to go, and shouldn't.