Originally Posted by
tomsteve 
im not the smartest, but it seems like since you have kids together, it might be wise to start now on legal action necessary to legally get and keep him away.
This. As a parent, he has certain rights. Depending on where you live, without a custody/visitation order in place (which you need even if you weren't ever married), there might be nothing to stop him from picking up the kids and refusing to return them. Moreover, he has an obligation to provide financial support for his children.
Please get some legal advice ASAP for your own protection and that of your kids.