That reacting is terrible aye? I got better at not doing it over the years and got to a point where I didn't even flinch when the XA started destroying stuff round the house - never his stuff I use to notice too.
One good response (that wasn't a reaction) I used often was saying "I'm sorry you feel that way." Then I would go find something out to do in another room, if he followed, I'd go sit in the toilet.
It strikes me as a rather crazy way to live now but it actually helped decrease the dramas.