Well, you were specifically talking about your son being "talked into" staying longer at Dad's than was planned/scheduled. IOW, not times when you had collectively PLANNED for him to be there. I get that he's a responsible kid, but that cuts two ways--yes, he takes responsibility for getting himself where he needs to be, but he also takes responsibility for his Dad's feelings.
It sounds like he's spending a lot of time there anyway, I don't think you are depriving anyone of anything if you put the kibosh on Dad's wheedling extra time together (time that appears to be distressing your son). I don't think it's out of line for you to say, "We agreed you were going to stay x days at your Dad's, now it's time to come home."