Hey, just be honest, every day visits weren't what you had in mind, you were just trying to be nice and neighborly. This is taking advantage. It is not a community pool. You call and ask first, every time. That's being nice and neighborly.
And absolutely speak up about the roach and not wanting your son to be exposed to that.
I know you're afraid that if you ask them to cut down on stopping over and to call first that they will stop coming all together and become standoffish and this is probably true. People are just like that, no matter how nice you are.