Sounds to me by his behavior that he is calling your bluff. So whatcha gonna do?
You have no reason to feel guilty. He created this toxic environment for you and your children. You just happen to be allowing it to continue. As far as him threatening you ..... He's delusional. Whatever...
Time to take control of YOUR life. Do what you have to do for yourself and your children and don't look back. They will thank you for it in the future, no doubt.
Keep coming back. We are here for you! Big Hug.