In my experience it is best to be clear and firm. Not rude, but no weaseling. "Thanks but that's not a good idea." "I appreciate your offer but I have to decline." End. No maybe sometime later......and you can also say "I don't want to discuss it further."
These guys will take notice if you waffle- and will gladly take advantage of any sign of weakness. Just my humble opinion.....sounds like you guys got this