He's a controlling man who's having a hard time giving up control of you. That's my interpretation. Please be careful, because this is stalker behavior. Find out what grounds are illegal enough for you to be able to report him to the police. I would start a journal of when he's driving by and any other details. It's better to be paranoid than surprised in this case.