It sounds like you are making the very best decision to take care of you and your recovery.
I would say something now, while the strong feelings are with you. I would simply say that I won't be involved in family vacations anymore as I will be doing things differently now.