Leaving a job is a lot different from leaving a marriage. Hopefully you don't have an emotional attachment to your job--at least not the way you would to a spouse, partner, or family member. And if you WERE in a marriage and said you were miserable and staying only for the money, I think we'd be urging the same thing--not worth it for the money.
You spend a huge portion of your life working--to me, no amount of money makes it worthwhile for me to stay where I'm truly miserable. If I were you, I'd be trying to explore alternatives and how to swing it financially.
But it's your life--if the money IS that important to you, then I guess it's worth it for you.