I agree with Anna.
If this guy would kick you (& your kids) out over this, is he really worth hanging on to?
He's probably contributing to your self esteem issues.
Counselling helped me a lot with my self esteem issues, but you need the right counsellor.
I'm not sure what you do with your days but volunteering on my community really helped me feel worthwhile and help me feel I had a sense of purpose.
Both those things go a long way to improvising self esteem, I think.
I also surrounded myself with people who supported me and told me it was ok to be me.
D