I would keep inviting him to join you for coffee. Personally, I was entirely too exhausted and raw in the first few months of sobriety to do much more than get to a meeting. I felt OK with my sponsor. That was all.
The fellowship stuff was way overwhelming.