rozied, It doesn't matter how old our children are, be it 8 or 58, their still our children. So yes we still worry, we still want to see them healthy & happy. Once they reach adulthood we need to let go, we need to focus on ourselves.
Go easy on yourself, if visiting your son is something you are not ready to do, then don't.
Like Julie said, write him a letter, explain to him just how hard it is to see him at this time.
Above all else remember he is an adult, and you need to take care of you. Take back your life.
Hugs,
Chris