I think if you're able to protect yourself and the kids and enforce what you have in place, I agree--make him file to change something, and then jump out with guns blazing.
If he gets no response other than no visits and can't tell if he's getting to you, it's possible he backs off. It's a lot of WORK to successfully go after you in court...that's not nearly as fun as harassing you.
Unless of course anything elevates. And I know you know to document document document.
Ugh, I'm so sorry!
Can you deal with the continued nonsense? Your attorney may be hoping to spare you the drama, though I agree it seems unlikely to end it (if he insists on giving you drama).