RED FLAGS - RED FLAGS - RED FLAGS
Trust your gut, this is NOT the man for you and especially NOT the man to have in your children's life.
Do yourself a favor and end this relationship, work on your codependency and never ever again settle for less then what you want and deserve.
We codies often keep trying to fit that square peg into the round hole then wonder why we are so unhappy in our relationships.