WTBH - I don't know if you remember but I'm in a similar situation. We have two children and live in the same VERY small town. Most of his friends live on my block and one of them next door.
How long is he staying for each visit? Is this something he's requested or something mandated by the courts?