I think your reaction to it matters. I'm not sure, though, that statements like that are necessarily "red flags"--some people just fancy themselves extremely intuitive and think they always know what someone else REALLY means.
Come to think of it, you're sort of doing the same thing, aren't you? Drawing conclusions based on your perceptions?
I'm not saying you're wrong, and even if it's mean innocuously, if it drives you crazy that's legitimate.
Maybe a break to think about how you feel about him and about your relationship with him wouldn't be a bad idea...