He has no more right to just walk into your home than he has to just walk into a stranger's home. He does not live there, has never lived there, so you can call the police if he continues to do this. Yes, by all means, speak with your attorney about this.