Personally, I think you can't go wrong by simply sticking to the letter of the parenting plan. Both kids have NOT requested to go visit Dad there, so it isn't happening. Period. All you have to say is that he's welcome to visit with them here, but your younger son has plans that don't include a visit there. You don't have to elaborate on those plans--the thing is, he hasn't ASKED to go visit dad there, which was part of the deal.
When you talk to the kids, I'd suggest you lay that out as their "out" if they truly don't want to go. And I'm with you, that there should be ZERO pressure on them to do that.