I don't think serenity means being a doormat! Seems like it's best address the things that negatively impact our serenity otherwise we won't have any. Good on you for standing up to this bully!
I'm with Pelican - perhaps this incident will usher in the era of a better neighbor relationship with this guy. Even if that means a civil distance. I kind of look at things from an animal training point of view; if your (neighbor) (dog) (cat) does anything positive, reward the try.
Eventually, it becomes habit for them.
And don't judge yourself too much for this wee event in your life. You did good, no more and no less than what you needed to do at the time. ((pat on back))