I agree with you and feel the same way. The other day, New Guy complimented me and I just said thank you. Then I was like, hey, I didn't question him or blow him off or wonder what he wanted from me.
Like others have said, it's the actions over the long haul that are the most telling.
He understood and has been totally decent about me saying no (as I would expect). We still chat now and then and he continues to be complimentary in a sincere way and told me that if I change my mind about dinner, to let him know. No pressure and it's nice...
Ditto with me and NG. No pressure at all, just an open door. When you're ready you can walk through it, knowing you have the wherewithal to walk right back out if you don't like what's inside.