Stay strong for yourself and your children!
Do not let me scare you with threats about getting custody of the baby. That is highly unlikely. See an attorney to get the facts and ease your mind. Since you aren't married, you have it much easier really. You can make him go through the expense and time of proving paternity if he wants to have any visitation with the baby.