I am worried about a scenario when they all drink and I feel too tempted to turn it down.
Grind,
go with that. it doesn't matter , ultimately, what they would or would not do or say or think; what matters is that you're not confident. trust that. be safe.
when i first got sober, there were events i could attend and some i couldn't.
for some events where alcohol would be present i could make plans (like driving, sticking to a non-drinking person, having a soda in my hand) and others where i just felt a real nagging doubt...i decided the benefit of the doubt would always go to my sobriety.
it gets easier.