I don't have much to add to what everyone else has already said but one thing. A couple should share their experiences of the day with each other. It may seem insignificant to you right now, but it's communication. If you are being as secretive in your communication with him about your work on a regular basis, not being willing or able to meet with him for a lunch date (which I happen to think is sweet of him to ask) and then all of the sudden you are throwing a "party" for a guy in your office...call me crazy, but that would be a red flag to me too! Most affairs start in the work place...need I remind people of this?
I am by no means saying that he is right in other areas...you are certainly not the cause of his drinking...at all! It sounds as if he doesn't know how to express his fears to you...it's not just the Code's that have abandonment issues...Just a thought!
Good luck to you!