Great big hugs to you. It's funny, but all of the activities you did with your AH are things that we've done and found connection from: doing jigsaw puzzles, playing badminton the lawn, etc.
My AH is currently in a depression phase, as well, and I am feeling much like you are. I just started a thread about my codependency rearing it's ugly head again, LOL! The only thing I can suggest is do what I'm doing: take care of you. You enjoyed those moments with him and yes, they were real and he probably enjoyed them too. He is an alcoholic and they suffer from issues with shame. My AH used to tell me that he felt guilty if he enjoyed anything because his own guilt and shame would kick in and he'd think he didn't deserve the enjoyment. He'd have to force himself to enjoy something and we lived like this for years, but I was truly blind to it for so long. So, I don't think your experiences with him were false, he just can't let himself feel the real feelings it created. They live behind mirrors, behind false selves, and that in itself doesn't allow them to truly have joy in the here and now. If they do, it is fleeting and their perspective becomes distorted by the alcoholic thinking. It tells them something different, and they start to believe it. I may be totally off base here, too, by the way, but this is how my AH has told me about his experiences(only when he really is in a communication phase with me, which he isn't right now).