Wow they are being incredibly rude! Talk about taking advantage.
I agree with what everyone else has said. This is your property, your home, your liability, likely, and most of all your peace of mind at stake.
You said:
"I want to withdraw this open invitation I have somehow created. I want to reclaim our privacy."
You have every right to do just that! Be firm, direct and honest. Don't apologize. They are the ones who should apologize. If they are bent out of shape or annoyed you can live with that. You are not being unreasonable. You made a mistake by being too generous, but you don't have to keep paying the price. It's your home.
Certainly don't let your annoyance over this tempt you to drink.