My dad died when I was six, he was an only child so my grandfather loved it when my brothers and I sent him Father's Day cards and gifts. It helped him on a day when he missed his son, and helped up on a day when we missed our dad. Sad as that sounds, it was always a happy day.
Please let your son find happiness sharing with your dad. It will help the boy feel happy and part of a special day.
Hugs