Trust your gut and your instincts about safety. If you're afraid of him at any time...'listen' to that. Maybe he hasn't outright beat you, but he's been physical with you; he's laid hands on you. Abuse tends to escalate and often starts as verbal and becomes worse over time.