I'm sorry that your son is being manipulated too. What a craptastic thing to say to your own child that you might just miss?
Another spin is that XH is trying to push your buttons so he can open an argument and attempt to vindicate himself...he does NOT want you to be happy or peaceful, he thrives on controversy, he might feel that he can "win" at some point because he imagines that your son is interfering... Of course you have wild sex with several men as soon as your son steps out the door.
(this was the mindset of my own AXH for a few years, hopefully yours is more rational).