Toss and Anna, I'm so sorry to hear about both of your dads. Losing a parent under any circumstances is a really hard and emotional thing to go through. Please know we're all thinking of you! Sending prayers for strength to cope.
Toss, I think your plan sounds like a good one. You made an effort when your dad was alive, and it doesn't sound like being there with people you don't know would be helpful to anyone. Visiting the grave at a more private time in the future with your own family is a nice way to honor your father and keep your own sanity. It sounds like your brother understands.