It's hard to make conversation with children. I agree that some sort of activity would make it easier. If you went to a movie then had ice cream after you'd be able to talk about the movie while eating your ice cream. If you played miniature golf you could talk about the game you were playing.
Also, if you can get him to open up about anything, really listen and you can later bring up the people and subjects he discussed. You'll build a history with him. In time, he'll get used to you being an important part of his life and things will become a lot easier between the two of you.
IMO, you're doing fine. Despite your uncomfortable feelings, you're putting your son first and showing up for him.