I learned something once, probably from my mom, she's the best, if you have to say something bad or difficult, say 3 things nice first.
So: "I really like that our pool is a nice place for your kids to play, I am happy they enjoy it so much, it's nice to have neighbours that are also friends, that said, I need to ask that you cut back on your use of the pool. I have found it to be really distracting to my work which I didn't realise when I opened the invite and I also would appreciate being able to enjoy the pool privately just myself and my husband. Going forward please ask me in advance if you would like to use the pool for a quick dip. I also have to ask you not to use the pool at all if I am not at home, it is a liability issue and my insurance company won't cover me should anything happen, I can't risk that, and it is not a good idea for you either."