Then I'd say, please drink out in the open so that there is more honesty between us. There. Done. More will be revealed as to how much his drinking affects your lives. It is odd though, that he labeled himself as a recovering alcoholic when you met and now it is turned around to he hid that because "you thought it was bad"....hmmm?
Not listening, forgetting, or not committing to memory ARE consequences of drinking, and i can attest that the sober person remembers the conversation with far more accurate detail. Have you ever had the exact same conversation over the course of a day, several times, with him acting as if you never had the conversation before? That's something that heavy drinkers do.
But, that said, that doesn't mean that he simply CHOOSES to not listen, commit to memory, etc. In otherwords, this bad listening may be related to drinking, may be a choice of his, or a combination.
A bad memory is a side effect of drinking, for all, alcoholic, or not.
Tying physical intimacy to emotional intimacy--not separating or compartmentalizing them--seems healthy to me, even if he didn't like the results. Explaining to him how you don't separate physical intimacy from emotional intimacy and that the two go hand in hand might give him insight if he is the type that can compartmentalize. This has to be explained to him without accusation, or he gets defensive, and there are then no lightbulb moments. Otherwise you could receive the common "you withheld sex as a punishment" response, which gets the marriage nowhere.