I suggest you continue NOT interacting with him. He is a very unhealthy man.
As long as you don't break any laws you don't have anything to worry about with the police. If he keeps involving the police with bogus calls he will soon regret it, they don't appreciate that kind of stuff. If you are innocent and he has no proof of you being there you don't have anything to worry about. (Is there even any legal reason you aren't allowed to be on his street even if you had been?)