Nobody thinks you are dumb, and I totally understand not being ready to report it to the police for a criminal complaint.
HOWEVER, for your own safety, I would strongly encourage you to file for a restraining order. The only consequence to him would be if he were to violate it. (I'm assuming he isn't a police officer for whom there could be employment consequences as a result of the order.)
And, as the others suggested, if you are unwilling or unready to report it to the police, do have someone documenting what is going on, so there IS a record.