I don't think you are wrong at all to not want to see him. Physical abuse is a zero-tolerence offense in my book, and you absolutely have the right to just ignore the email.
If you feel comfortable enough you could allow a written amend, but as you said he's still probably charming ( aka sneaky ) and might still have alterior motives behind it.