it would seem reasonable that if he wanted to see his son, he could have come on over and joined in. however, maybe he already had a drink and therefore couldn't drive. or, maybe he wanted to have some drinks, and felt uncomfortable doing it with you and/or his familly there. plus, if he had a few, how would he drive himself and son home?
i feel you did the right thing honoring your sons request to stay where he wished. it's not as if you excluded your ex from seeing his son.
naive