Well, I wouldn't talk to him, if it was me.
Otherwise, I don't know. He's likely to retaliate if he knows it was you, but he will think it was anyway. My gut says stay strong and stand your ground. Sometimes they do end up running away with their tail between their legs once they realize that you're not going to take it, any of it, anymore.