Am I being dense Gneiss (it happens quite a lot lol)
Never mind the gf and the kids why is your ex staying with you?
You are not over him and you have a history of drug use together.
If he wasn't still in your life she wouldn't be in your life.
Sorry to be a bit flippant but don't tell me he is a nice guy when he isn't completely taking advantage of you.