Yup, not going is 100% the right decision.
COVID is a perfectly handy and valid reason.
Like others have said, I really wouldn't entertain any discussion about it. If anyone wants to press, you can just do the broken record thing as often as anyone chooses to press. "Sorry, but I've made up my mind. Hopefully next year will be different."