The one or two guys that were drinking have to go, or be suspended. If you were accused and are innocent I would set the client straight. And fast. That's a serious allegation.
I owned my own contracting business. With as serious as my drinking got, I never, not once, drank on or before going on the job.