a simple "thanks, but it's not for me" is all you need to say.
I'm sure they get dozens of polite refusals every day, & probably a fair few not-so-polite ones too. I bet they won't bat an eye & will have forgotten about it in two minutes.
there's no point giving them free space in your head when they will almost certainly have put you out of their heads by the time they're two houses down the street from yours.