I remember leaving my first husband when I was still pregnant with my daughter because he was physically and emotionally abusive. I couldn't seem to do it for myself, but I had no real trouble leaving for my daughter. i didn't want her to grow up with that kind of drama either. She had a peaceful childhood with me and I know it was tons better than it would have been had I stayed with her father (years later he did serious jail time for child molestation to a step daughter, in his 2nd marriage). God was obviously looking out for my daughter then, since he never saw her again after the divorce.
You did the right thing. You both deserve peace without constant drama and upheaval.