Go with your gut. You say right in your message that you'd feel a lot better not going, so don't go. And in the grand scheme of things a work party is pretty low on the "ladder of importance". How about finding a community holiday meal or food pantry to volunteer at for a couple of hours instead?