He is bringing nothing to this relationship and you are doing all the work...that itself is a big red flag.
You deserve better, actually you deserve to be happy and have a peaceful life with or without anyone else sharing it. You alone hold the key to your happiness.
Your children deserve a happy and healthy home. They learn from what they see, so maybe let them see a strong mama who can live life on her own and who is selective on who she lets share it.
Hugs