Would echo Fini's post, I'd feel uncomfortable too, and would see it as an invasion of privacy. He wasn't invited to read your private correspondence, much less use that information to connect with you on social media. I'd find that creepy, and crossing boundaries if that happened to me, and it's the not having offered the information that would bother me most.
And yes, have that conversation with him, about your not being comfortable, and why he thought this was OK. It can only help with how you decide either separately, or between you, how to proceed.