Brynn, please listen to that voice that is telling you to leave. And, if you are afraid he may have a reaction if you leave, can you leave while he is away, as others suggested, or have a friend come over to help.
One thing I've seen in my short time at this is that others can get very upset when we draw boundaries. That doesn't mean what we're doing is wrong. On the contrary, it means we're doing precisely what we need to be doing.
My thoughts are with you...