He never hit me, only shoved, pushed, yanked, and hurt me.
This is not acceptable behavior from anyone.
If a complete stranger did this to you -- would you find it acceptable?
No.
Then why do you question if the person who is supposed to be closest to you has the right to treat you like this?
Keep running from this man. He is dangerous. And the "I'm not easy to live with"? That's a symptom. It is not your fault that he hurts you. Ever.