Or maybe we just think we have to justify not taking it.
I don't. I explain if I feel like it. Otherwise, "No thanks" is more than sufficient. If the person pushes, just say, "Because no thanks," smile, and go about your business.
People for whom alcohol isn't a big deal won't care that you aren't drinking and people for whom it is aren't there to listen to you anyway.
d