Wow, that would drive me crazy. This is definitely a situation where you wish you could take back your initial invitation, but, you can't. I would go for being nice, but honest. I would tell them you didn't realize how noisy and disturbing it is for you, working at home, when people are in the pool. I would say that it's affecting your work more than you imagined. Secondly, the liability issue is huge. I would expect you would be in all kinds of legal trouble, not to mention emotional, if there was an accident in your pool and you weren't there. You could mention that it's too stressful for you to consider what might happen.
Whether or not that would work and you would remain friends, I'm not sure. But, I think it's worth the risk for your peace of mind.