Since you have repeatedly tried to cool things off with him and he has rejected the premise, no, I don’t think cutting contact without communicating your intent first is at all rude or immature.
You always have the right to leave a place that is making you uncomfortable—a party, a room, a job, a conversation, a relationship. You do not owe anyone anything when enforcing your own boundary.