My retort would be, "That is correct, you don't ever know when someone may go. However I have plans to spend with grandma another time because today does not work for me. I love her and will definitely be spending time with her another day." Polite and direct. Also if you could say you have already told grandma yourself you will not be there it would remove everyone else from the equation completely.
I did sit my family down last year and explain to them (they spend alot of time together and have lots of functions) that while I would love to (and I would) that I cannot make every single thing they do together. Sometimes it is just because I am busy and need a break for myself. Sometimes it is because we have another side of the family we also spend time with. I found that conversation really helped them and me. It made them stop pressuring me to be there and then I did not feel guilty for missing things. I do still spend lots of time with them (they are my rock, love them), I am still free to say that I just cannot that day.
Explaining the method to your madness can go a long ways lol!