Do you have a family member or friend who could take the kids to a place where your ABF could pick them up? Then there would be no reason for you to see him or talk to him.
If you know he's still using, though, I would recommend supervised visitations. Maybe a friend or familty member could supervise the visitations?
Good Luck!
Sue