I'm not really clear if you feel in physical danger or not either. He sure wouldn't be the first person to take a drive past an ex's house when feeling blue or whatever. I still drive past my old house sometimes (ex doesn't live there though). Of course he wouldn't be the first psycho stalker either so just to be safe I'd make sure I had some kind of self defense 'plan'. Pepper spray or panic alarm or something.
I also think it is OK to tell his family you no longer want contact with them. There is no point in it really. His treatment is his problem and I'm quite certain he can get excellent care without your input - to think otherwise is just dramatic on the part of his family. If they are telling him they talk to you - that is just feeding into him driving past or getting crazy thoughts etc.