Really, you can figure out all of the nuances of your own behavior once you have a bit of time and space.
What's keeping you there NOW? Given that this man has physically threatened you (in earshot of your kids, no less), what's it going to take? A black eye? Broken bones? What if he hospitalizes you? Could you trust him to look after the kids? Or will CPS have to get involved?
What would you do if a friend was behaving in this way towards you?