Does your parenting plan state that he is to have an opportunity to "make up" a day? Does it dictate that you agree to be flexible.
My sister WAS held in contempt for something similar, so I hope you know the order like the back of your hand, as unfair as it is. Just make sure all YOUR ducks are in a row. And documentation of "he forgot". Because in my opinion, him "forgetting" does not mean that you have to flexible to his memory or obligation to children. Good riddance!