I think a lot of us focus on the question of "Is it bad enough to leave?" rather than "Is it good enough to stay?" I know I didn't leave until it was "bad enough" and sometimes I really regret that lost time... But it's a part of me, now, and where I came from is a large part of the wonderful person I am today.
That said, the decision really is up to you, and you are not a bad person for hoping he'll just do something to make the decision easy. That's what I waited for before I left, and I do believe that I am not in the minority on that point. It's easier to say we left because of someone else, it's their fault... But there is nothing wrong with saying it was your decision.
If you choose to stay, make sure you have your boundaries in place to keep you safe. Trust once lost needs to be earned, and he needs to understand that it's not going to be simple or fast.
If you choose to leave, make sure you recognize that his choices are his alone and you are not to blame for any choices he makes, even if he says you are.
Either way, you can change your mind at any point. You don't have to justify your decisions, and you don't have to stick to the same one for the rest of your life either. Whatever you decide is not set in stone unless you decide it is. :-)