This is why you need that order in place. No negotiation necessary--the order is the order and if you stick to it without deviation nobody has any grounds for complaint.
This just sounds like a never-ending painful exercise. I like the idea of thinking of some standard, non-negotiable responses like, "Sorry, that doesn't work for me." You aren't obligated to explain WHY it doesn't work for you. As long as you aren't depriving him of all access to his child the courts will understand that one parent can't call all the shots for their personal convenience.