what you CAN control -- your own decision, go or not go.
what you CANNOT control - what HE does, if he goes, what he does where he goes, and how he feels about you going or not going.
if he can't handle going to see family, that's his problem. if he isn't going to go SEE his family, cuz you choose not to go, that's HIS problem.