I have the same question about whether they are a heavy drinker.
You were smart to leave, you did what you needed to for your sobriety. If that happens again and the person is continuing to badger you, a firm no, and please do not ask me again is definitely reasonable, and so is walking away from the conversation completely.