Good memories are what you make them. If it's not possible for your kid to go fishing, make a different fun summer memory. Catch fireflies in a jar. Bury something silly in your garden and send him on a safari.
My boys are 22 and 24 and grew up in an unstable and sometimes unhappy home. Today, their happy child hood memories are about the times we laughed until our sides hurt, card games we used to play, etc. They were able to have some fun and learn about some cool "guy stuff" when they spent the weekend with friends who had "normal" dads. And, as young adults, they are filled with wonder and a sense of adventure. They make take up fishing NOW. Who knows.
Enjoy your time with your son, and try not to dwell too much on what he's missing out on. He has a terrific mom who loves him very much. What a lucky kid.
Hugs
Cats