It is rude. If they would just ask and accept your no, it'd be ok. But pushing it is rude. Not sure if anger is the appropriate response though. These things will happen. You can just accept that, feel calm and be assertive. No one can force you and if it means they get offended, you know you're hanging out with the wrong people.