Seeing as it's 4-6pm couldn't you just say I don't drink during the day?
I've been told that it's 'vulgar to drink by the clock' but my mum always had a strict 6 o' clock rule.
Chances are though that no one will notice if you're drinking or not.
I've had to do work things with booze around and I did have to mentally prepare myself. It got a bit weird when people kept on offering me wine but I just said I'm not drinking or started asking other people if they wanted some.
I think if it's a networking thing it would be a shame not to do it because there's booze around but I'd imagine that having to do it every week would be a bit much!
Hope it goes okay x