Just a suggestion but when your Son hightails it out of the room (smart boy) you could ask the ex to leave. Why he thinks he can then just hang out with you I have no idea.
A simple - the visit is over - should suffice.
Also, I would shut him down the moment he discusses anything other than your Son. As soon as the words - You and Brother starts to leave his mouth I would be walking out to the kitchen, no reply.
Don't JADE - justify, argue, defend, explain
You will never get him to see your point of view, that's obvious at this point, it is impossible for him, you are of two separate mindsets, it just can't happen.
I'm sorry you going through this. Do you attend counselling of any kind?
As your Son is an adult and the visits aren't pleasant for him it would seem, perhaps they could be cut down to say once a month?