Forgive me if I'm wrong, but I detect that you are feeling guilt becuase of the fact you listened to him.
Just because you shared the same space does not give him the right to harass you. PERIOD. Whether you stood there or not.
I agree with the person who said that you'd already given this nutball enough chances to do the right thing...but that's the point isn't it? I mean you've drawn your boundary, informed him of it and he's decided that the rules don't apply to him.
I think HR needs to be informed immediately. If your going to set boundaries it's up to you to enforce them. If you don't he's going to keep this up, possibily even escalating!
Go to HR. You have to stand up and show him his behavior is NOT acceptable.