Controlling people really ruin it for me too. I'm pretty laid back until they start. I also tend to react like you and just stop going for a while.
I hope others here will have some ideas. I only have this problem with people I haven't previously encountered, to be honest. Once I know someone usually acts this way, I just ignore them and if they try to talk to me, I move away without engaging. I just say, "Excuse me," and walk away. It's not worth it to me to waste energy on them. I imagine if everyone does this, they may get the hint.
Probably not.