I'm sure you don't want the expense of it, though perhaps there wouldn't be a charge, but why not call your attorney and get guidance on how to handle this insane situation?
Could it be that if he fails to show on Friday that he forfeit's Saturday as well? Seems to me there is a "reasonable" clause within all divorce implied or explicit. Is it reasonable that you go about your day Saturday without plans or having to adjust them constantly because A-hole picks up the phone and decides "Now"? I don't think so.
I agree with Bluebird that you have got to put your foot down on this. Inch equals many miles with this guy. Find out your legal rights, and if it means he forfeits visitation then so be it. Your children will be ok with a couple of times it won't cause them any issues long term to not see him a schedule weekend or two. No, I would absolute not accommodate him to see her on Sunday. NOPE. This is the problem my friend, he preys on you and the feelings of your children. You do what's "right" in your mind for your kids, but its all wrong as far as what you get back out of it. I'd adhere to Fri and Sat as is in the decree. Until he shows that he can act like an adult and be amenable both ways in scheduling and accommodation I wouldn't budge one minute.
And if it were me ....I'd be changing the location of the party. If he has done it for two years you might as well write out a place card with his name. Maybe a friend would be willing to host, or move it to a kids place or even a park. Anywhere but where he can find you.