I'm not familiar with the law where you live, but in most places a 16 y/o is allowed to decide whether s/he wants to spend time with the noncustodial parent. Under the circumstances I cannot imagine a judge forcing him.
At this point, I think I would just inform your ex that your son doesn't want to see him. Period. You don't have to tell him why. Let him go to court if he wants to push it, and my bet is that the court will be on your side on this. Talk to a lawyer if you can, but I'd let your son off the hook for these visits that are upsetting him that much.