Instead of wishing for him to die (I know you don't really, really mean that), why not just leave? You aren't married to him, so it's a lot more simple than if you had to go through divorce.
It just seems to me that if we've reached a point where we find ourselves secretly thinking that we'd be free if he died, that it's time to find other, less drastic ways to get out.