Sounds to me like your decision is made, it's accepting your changes in life that is difficult. Friends change as we change. There are many friends I don't have anymore because my family and sobriety is more important.
With growth, comes sacrifice. We don't get it both ways. I hope you can find peace with saying No, and being comfortable with that choice.
Good luck.