I've been doing exactly the same for a number of years now. I simply don't show up. And with regards to this, I feel comfortable with the fact that it doesn't bother me what they think of it. As you point out, maybe that's where you need to get to as well - not being so very concerned of what others actually think of you/ your actions/ lack of actions. It's not selfish, it's attending to your own needs.