You aren't paranoid
Your concerns are very legitimate. My ex-A is no longer allowed to spend time with our son when I have the slightest indication that drinking is going on. I feel that my son is not safe with his father when said father is drinking. Those are the rules I established after the last binge. So far, ex-A is healthy and sober after spending some time in rehab. But if I ever get clue one that he is drinking again, he will not spend time with the boy. Those are my rules and I feel good about them. I hope this summer works out for your kids. Maybe you can work something out with your oldest to check in with you and let you know how their Dad is doing.
Peace,
Gabe