You did the right thing.
I had a similar experience with a guy who was calling my father names.
When I had the chance to confront him face to face (we had already had words over the phone) I just stared him down, did what I had to do, and left.
I think he knew what was up, and I didn't get arrested that day.
Feels crappy sometimes, but the right thing isn't always the easy thing.
Be proud you aren't drinking over it.
Who knows, maybe the guy has his own demons, and thats why he acted like such an ass.