You're not even married to him. You have no obligation to this guy. You have no requirement to stick around and see if he changes.
What makes YOU happy? You have every right to follow through on that answer.
So what if someone else is willing to put up with his behavior. Are YOU?
Also, abuse doesn't have to be physical to be abuse. There's also emotional and verbal abuse, which is what my XABF loved to send my direction.