I'm with Anna.
For the big bash our owners through at the restaurant on Thanksgiving: "I'll be spending the day with my family." It was true, simple, enough said. I also refrained from bringing the topic up with others (just to make conversation) so I was involved in less of this than I could have been.
I also have no problem lying - if I hadn't had plans with family, I'd have said the same thing.
A simple response is sufficient- you truly don't owe anyone an answer for this kind of thing and - honestly, truly, REALLY, people will talk about you less than you think. The ones who keep batting your absence about don't have enough to do, perhaps a drinking problem of their own (if that's a big part of this event and/or what they used to do with you) or some other motive that you're better off staying away from anyway.
You can do it.