I also find ego at the bottom of my defensiveness, momentary irritation, flashes of anger.
My true self is calm, accepts people do what they do (just like me) and doesn't easily take offense, or go on the offense.
That doesn't mean I'm a doormat--but my way of saying "no" is increasingly not aggressive or coming from fear.
That typically will de-escalate a difficult situation instead of increase the problem.
I'm much more peaceful, in short, when I step back and observe my ego instead of responding from it.
Not always easy, but noticing and making the disconnect is the first step.