Hiya
Six children on your own must be tough, I can see why you asked him to come back, and his refusal certainly must have seemed like a slap in the face.
I think in his mind he now thinks he has you where he wants you, lonely and needing him.
Maybe distance yourself from him entirely for the time being, dont reply to his texts etc, or discuss anything not related to the children. Tell him cheerfully that you think he is right coming back is not the solution and thank him for pointing that out.
He has somehow managed to get everything he wants now (in his mind). He can see his kids, drink when he likes and thinks he can come back to you anytime HE gets lonely.
You have us here to talk to.
Best of luck hun x