Can you put some time limits around Christmas dinner? E.g. you and your son have to be somewhere at 7.30, so you'll need to have everything finished by 7.00? Possibly a friend could help you out by offering a well-timed invitation.
From what you said about your ex, it sounds like he's a little undisciplined about paying attention to when he is or isn't in someone else's space, so having some external boundaries on the dinner might help.