I do this a lot.
Someone glances my way and I think: "**** - why did he/she look at me? Did they see me do something stupid when I was drunk? Do I look like **** today? Did I sleep with him and not remember? WHY THE F*** IS THIS PERSON LOOKING AT ME?"
By the time I have gone through this thought process, the person has long stopped looking at me.
I'm trying to look at it more positively - maybe they think my purse is cute, maybe they noticed I blew out my hair today etc
But often, I suspect, the truth is what was said above - they glanced at you but didn't even register it because they are too busy thinking about their own lives.