Jessie, hon, you HAVE to do what is necessary to protect yourself, even if you look goofy doing it. I'll tell you a short story. There was a person at work who I HAD to work with for quite a long time and very closely. She is TOXIC to me (Didn't take me long to figure that out). Other people may be fine with her, but I'm not. Whenever I had to work with her, what she would do and the way she would talk, and the way she worked with other people just made me SICK. I was stressed out for over a year having to work with her. I spent all kinds of time, effort, and energy trying to figure out what was wrong with ME. What did I need to do to get along with this person? Why don't we get along? How could I fix the relationship between her and I? I would read articles about getting along with people. Examine myself and my own shortcomings. Talk to people who got along with her and asked them their opinions on what I should do. Eventually, I got so sick, physically and mentally, that I had to bite the bullet and get another job, because there was no getting away from her if I stayed. For months after that, whenever I would see her, I would HIDE. LOL, yes, funny right? One time I saw her walking in from the parking lot, and before she could see me, I RAN across the parking lot to avoid her. That's how sick she made me. I am sure I looked weird running across the parking lot but that is what I had to do to protect myself. I did that for the next two years. Now when I see her I am OK and usually do not have to hide or run because I no longer have to deal with her at work, and I'm no longer invested in the relationship.
Next time this guy shows up where you are, RUN. Do not talk to him. Do not stop to explain yourself or apologize or anything. Just RUN.