Don't let his lawyer scare you. Look at what the plan says, if it says he can't make up the time, then see him in court. Save any evidence (texts, e-mails, voicemails) where he says "I was out of state and forgot I was going to have the kids" and any sane judge will say "well it seems your priorities are out of whack, Sir." End of story.
My ex threatened with lawyers and taking me back to court at every turn, too. Lots of noise and no action.