Stung, I'd say it's never too late to bring it up, as long as you do it sensitive to expectations. If you still believe he will treat is as a joke, the ask him how he would feel if you were to say the same thing to the kids about him? Jokes make people laugh, no one is laughing at this behaviour. He won't change without knowing he is doing wrong in the first place