I think it's a lot to expect the kids to have to tell him he's disrupting their schedule. They shouldn't be put in that position. They can tell him that after the fact when they do finally talk to him, if he demands to know why they didn't answer when he called before. "I was busy, doing x, y, or z." If he berates them they should be totally permitted to say, "I'm not going to listen to you yell at me (or say bad things about mom or whatever). Goodbye."
And YOU should not have to talk to him AT ALL. He can email you if he's got something important to communicate REGARDING THE KIDS. Then you can read it when you're ready and respond or not as you see fit.
I'd send him an email to that effect, just to make the record clear that you aren't preventing him from talking to his kids. Don't debate it, just state the facts.