If your son is "getting talked into" staying longer at his Dad's, this is where you can step in and say, "No." I don't know what your order says about parenting time for him, but when you "let him stay" longer at his Dad's house, you are basically rewarding your ex's manipulation of him. If he were happy at Dad's house, then I'd be saying sure, if you don't think he's in any danger there, let him stay there as much as he wants to.
Your son might find it a huge relief if it weren't on his shoulders to say no--whether he admits it or not.