Everything that amy55 said. Both posts. I lived that situation too. I was also walking on eggshells and lowering the bar for what was acceptable behavior until my "normal" was actually really unacceptable to most sane people.
Just let this guy go.
The worst part is your son knows this is happening -- it doesn't matter if you THINK he doesn't know, he knows. Kids know. Surely he hates it. You are his role model. You need to protect you so that you can protect him.