Well, do you need to clearly respond to him in writing that it is not OK
and that he had better return the children and follow the parenting plan or face the consequences?
If you don't say anything to disagree, it could well be interpreted as you complying.
It seems to me you'd be on more solid legal footing this way,
but that's a question for Lexie I suppose. . .