Well, you and the kids are the ones who will have to live with it, and only you can decide what's an acceptable level of risk. You might be right, he might take this as a "win" and things will settle down.
Hugs, I know it's difficult to make these decisions, and whatever you decide, remember that you are making the best decision you can make under the circumstances.