I'm sorry you're still dealing with this ridiculous crap.
One thing--I don't know how the police work in your jurisdiction, but in situations like these we generally treated it as a civil matter. Even though, in our state, there's a charge for interference with custody, it generally has to do with concealing the children, rather than simply not abiding by the parenting order. Now the JUDGE who issued the order could hold him in contempt, fine him, etc., but in my state the police wouldn't arrest him and take the kids back to you (unless they felt the kids were in danger).
So you might want to call your local police department and prosecutor's office and explain what he's threatening and find out how it would be handled if he refused to return the kids as he's threatening to do, before you decide whether to let them go to his house. I think if you refused to send them on the grounds of the threat he made you would be well-justified in not sending them. Might want to consult a lawyer on that one.
It really, really sucks, I know, but he'd be doing the same thing whether you settled or not. On a positive note, it sounds like you've got the judge's sympathy at this point, which can only help, going forward.
Hugs,