I also think you are seriously under estimating yourself too, in that you won't be able to handle people drinking near you.
You have done an amazing job of being sober so far.
Why not go and try it.
Be open to it.
If it is really that bad, you tried it and you can know for next time that type of arrangement won't work in the future.
You can go to a coffee shop etc instead.
But how will you know if you don't try?