Sunshine,
You have nothing to feel guilty about! These neighbors, wow.. if their noses are out of joint now, are way out of line. I imagine them saying, "Wow, can you believe that Sunshine? She doesn't want us using her pool round the clock freely at our convenience, even when she is out of town, smoking weed and leaving trash.. Now she has the NERVE to only want it to be when we are invited and our kids can be supervised??"
You did the right thing. You've tried to be kind after making clear what you could not accept on your property. Now if they are going to be huffy and passive/aggressive about it, that's their problem. Don't let them stop you from using your own pool!
You sound like a lovely person.. please don't let this make you doubt yourself. You have been more than kind!