This guy is scary and he has been mentally unstable and scary since you first started posting. You are not crazy. He was dangerous then, and now, and will be in the future. If you want to protect your children, stay away from him. 100% away from him.
I would call the police and report it and find out how to get a protection order. Then no contact with him - do not respond to his texts. Call a domestic violence line to figure out how to be safe. If he comes pounding on the door don't let him and tell him to leave. If he doesn't leave right away or escalates, I would call 911.
Stay safe.