I'm glad you came around to realizing you are not responsible for his feelings! To me, it sounded like you were carrying way too much weight from a simple "no"! Unless you were leading him on big time (and I'm sure you weren't), you have nothing at all, nothing, to feel guilty about.
I agree that I'd be worried about getting involved with someone who threw that all back at me after a simple lunch. If his expectations were too high, that's HIS problem.