Ugh, I'm so sorry about your mother. Parents can be so damaging to their children's self-esteem, sometimes without knowing, but sadly, sometimes they know it and don't care. At least you're able to recognize that what he's saying is a trigger for you, and that you need to guard yourself against him.
Is there any way to minimize his presence in your life? Seems like you could use a break to regroup and strengthen yourself. I would also suggest daily affirmations, coupled with whatever grounding exercise you use (yoga, stretching, meditation, KICKBOXING!!). It takes a lot of effort to believe that we deserve to be loved, especially after years of being jerked around and demeaned. I'm still working on it every day.