Just reading your post triggers me... I know that fear, that inability to control "your world" if he chooses to do this, which he will.
Take a deep breath. He will be there. He will walk that line, pushing it just as far as he can. So what next?
Do you have any large, male friends? Heck, just any male friends? Will there be any parents of kids there that know your story? I think you'd be surprised how willing people will be to buffer this for you if you ask. And I'd ask if it were me, because that's a big burden to hold while doing everything.
I'm furious on your behalf. This is ridiculous and he knows it. And that's precisely why he's doing it...just to get to you. If at all possible find a way to disallow that. He can be there physically, but only you can open the door to your brain. (God, I know it's not just that easy. But you can do it.)
Don't know if you're in the mood for a laugh, but the first time I had to see X after several months, I was a mess. I told my friends I should hire a bodybuilder to answer the door in only a towel, looking like he just stumbled out of bed, and acting like he had no clue who X was or why he'd be there. Then to mumble something like "yeah, I think she mentioned she was married or something a long time ago. So...what do you want?" Ha ha, minimizing the entire marriage and the fact that he was a passing statement in my life (and I was "with" a hunk) would be hilarious! He might just implode and take his issues with him.
Didn't do it. But the thought still makes me smile.
I'm sending you major hugs and prayers!!!