I don't do things that aren't fun.
The exception is if the thing earns me money or helps another person.
This won't help anyone, doesn't pay a dime and you surely will not have fun. Actually sounds like you will be miserable.
You don't need any excuse. He can go without you if he would like. You are not responsible for his choices and if you were he would never have picked up in the first place. You are responsible for your own happiness though and this will not contribute to it.
The boundary for me is I will not attend an event where substances that cause me discomfort are freely passed.
I only feel guilt when I allow it. (I did last night but quickly got over it.)