so what IS it you are holding ON to? neither of you are happy....you can barely co-exist. considering your tenuous fractious relationship, it is not healthy for either child to be in the room with you at night. you've already acknowledged that the three year old is acting out.
he doesn't work. he doesn't contribute. he comes and goes as he pleases, while playing the sober card. you went so far as to seek relations with other men to get some momentary connection. you live with your parents, which could be almost ideal for now except for HIM being there.
when people show us who they are, believe them.