We codies tend to "awfulize" and worry about the worst. You extended an invitation and he accepted.
Maybe just let it go, walk with grace on your son's special day and let other people think/say/do whatever they want.
No fretting, this is a time of happiness for you. Wrap yourself in that and enjoy the day.
Hugs