Plain and simple. If he doesnt respect you enough to make changes. Obviously he doesnt respect himself. Then you need to cash in and move on.
You already know what you need to do. You posted it right there.
When it comes down to where someone makes you feel that way about yourself and them. You defunately need to get away from them.
Far from crazy and not alone.
Crazy if you stay.