I defiantly understand your worries, and perhaps anxiety over this.
It is hard not to go back and forth in your head about it.
I am kinda laughing(with all due respect) at some of the posts, because it sounds so bad not to want to live in the same town, or to not want to drive by the son's house that lives a mile away. To other people without this issue, it probably sounds horrid, but to us, it makes perfect sense!!!
My son still lives with me, and not for long, and I dread it when he calls, wakes up, comes home. I have often thought about skipping town.
Thank you for your post, and I hope it all works out for you.