Telling them you're not interested is not the same as decrying their religion or slapping them in the face. literally or methaphorically
I say that because I used to be a great people pleaser.
I used to go out of my way not to hurt people - I wish I could say it as always altrusitic... but often I could not bear the thought of people not liking me.
Thats not a good enough reason to spend time on things I don't want to do.
If you really think about it, like you said, you might be taking time away from someone who is really interested in what these people offer.
Set yourself, and them, free
D