You mentioned you aren't feeling too safe at the moment? Has he ever been physically violent with you? If not are you getting the feeling that he could be?
If either of these are true, I say don't wait, get out as soon as you can. Carefully, you don't want to trigger his anger.
If you are unsure about how to get out safely, you might want to call the DV hotline, they have plenty of experience with that. 1-800-799-7233. They also have online chat.
On their webpage they have some good tips you can read online:
https://www.thehotline.org/help/path-to-safety/
Talking to them or at least reading about a plan might make you feel a bit safer and be ready to act if you need to.
Originally Posted by
res01 
Am I considering throwing in the towel too early?
Not if you are finished with it, no. There is no rule that you have to sit around while someone abuses you to see if they might see the "light" and become non-abusive, none at all.
If you wish to revisit a potential relationship with him at another time AFTER he seeks help and is well on the way to recovering, you always can.