Ugh... I have a feeling this is going to be my life in the very near future. I don't have any great advice - but I don't think you're being unreasonable. If it was your weekend with the kids, and you had a preschedule event... that's what trumps... not his last minute whim. The one thing that comes to mind is your situation is EXACTLY why custody agreements, and all the nitty gritty details are sooo very important. Communication, communication, communication!!
I *just* got in a heated discussion with my STBXAH about custody. I wanted to discuss the details (ie. where would they live the majority of the time, how would weekends/holidays be split, how would we handle times when the one with custody is busy and needs a "sitter', paying for clothes/medicine, etc). STBXAH FLIPPED out. Saying I was being too pragmatic about the whole thing. He's so mad that I'm "taking him to the cleaners"... he wants 50/50 custody because doesn't want to pay a penny towards child support... oh, he went on and on.
And then he looked at me and said, "Don't you see how you are RUINING EVERYTHING?"
Me - "No. I'm trying to be proactive and have the details discussed so that we minimize misunderstanding and issues later."
If you don't have a formal separation agreement... get a lawyer and get one drafted ASAP. And if you have one, time to revisit it, modify as necessary, and start living by it. It sucks to be so regimented - but it really will help define things better for both of you, and especially the kids.