I am so sorry! I do think that he should know that is why you are doing what you are doing. It is a consequence your entire family has to deal with, he should share that consequence. I could hear in your post the other day what stress his not being able to drive the kids will put on you.
I too hate this. There have been so many times I feel like I have three children instead of two. I am slowly changing that. I am slowly making him step up as I need to see what sort of father he will be if we do separate. I too am scared, you are not alone. Take it a second at a time. Breathe. You are safe, your children are safe.
Have a safe trip!