Seems to me that he's using an awful lot of emotional blackmail to keep you feeling stuck.
Just because you have years invested with him, doesn't mean you should invest even more into something that is not working and makes you unhappy. Especially if he is violent.