.... that if I really cared about them I'd do whatever it took to ensure he had time with them.
Wrong. If he really cared about them he'd do whatever it took to ensure he had time with them. You are not to blame for his actions. He is.
I agree that email is a good way to go. Short. Direct. And creates history if needed at a later date. Maybe just reread a couple of times before hitting send to make sure there is no ambiguity.
What a stress this must all be for you! Do something extra nice for yourself at the first opportunity.