wantotbehealthy how petty and nasty of him. Here's a suggestion you could mull over which wouldn't require legal moves. You could write, or email him and basically outline that your job is in jeopardy because you can't be late. He agreed to collect them by a certain time; if he isn't there by x:xxam you will have no other choice but to take the children to school.
Be absolutely firm about not waiting 1 sec past the agreed time and diarise every lateness incident. If he goes to the trouble of taking you to court you'll have all your ducks in a row.
You're going to have to be pretty hard because that's all people like him understand. Good luck.