I second ALL of Tuffgirl's ideas.
Bring a friend with you. Only the most hardened of criminals and insane of the criminally insane would display that behavior in front of someone other than you.
If I recall correctly, you live in a small community where he is buddies with the cops. That' makes it harder, but not impossible. I know our children's services office does exchanges at their offices. In extreme cases, they will do the pickup and dropoff at custody exchanges.
But absolutely, have someone go with you. I've done that a few times with other divorced moms. I called the cops once when a father showed up drunk as a skunk and the mom broke and didn't want to call the cops. But that was the only incident.
And I strongly recommend a self-defense class, too. I took one last fall -- it was only a one-day class -- but you know what it did? It gave me the confidence that IF AXH attacks me, and he isn't carrying a weapon, I can take him out. I'm smaller than he is, but I know for a fact I can take him out. That knowledge in itself has changed the way I deal with him.
And that's the "fix it" part. Because that's where I go. To fixing it. I understand you're broken. I understand you're scared. But buckle up, Buttercup -- you're going to have to take a deep breath and not give up on this. Your kids need you. So find your inner grizzly mama and do what you have to do.