i think that the child and family services people might have some solution/ideas in this situation. this is a very common problem that i would imagine they see all the time.
just brainstorming, but what if his visits occurred in a place that is supervised automatically? that way, you could drop them off and pick them up (or arrange for a taxi or they could even take a bus)...
some ideas:
1. a gym. they could workout or take various classs. fitness instructors are there.
2. a public swimming pool. lifeguards are there
3. a sports team...like hockey or baseball or something.
4. a restaurant
5. the public library! he could help them with their studies and librarians are there.
6. the shopping mall
i think it is really important that you take care of yourself and your own recovery. if you don't want to see him or have him in the house, then try to make this happen.
these types of pressures wear us down if we don't take the steps we need for our own balance in life.
naive