As long as you say it in a nice conversational tone, then you are not being rude. You are simply stating your feelings on the matter. The other person can do what they want with the information conveyed.
" I have no intentions of joining their church, inviting them into my house for further discussion or converting to a religion"