Imma go a little stronger than the others, who all made good and a little "easier" suggestions....
If you want to stay clean
Don't go outside to smoke
Tell him he cannot be around you now
Warn him you will decide to call the cops if he keeps coming around
Call the cops when he comes over
My first sponsor told me "No." Is a complete sentence. This sounds like a fast way to get you using again so I don't think ANYTHING is too extreme to take care of your sobriety. Just IMO.