I can't imagine asking anyone I wasn't REALLY close to what happened to their marriage or relationship.
You might sometimes be reading more into the question, though, than they are really asking. It might be they are just wondering whether your son spends time with his dad--that's sometimes helpful for people to know, if only to avoid making any awkward assumptions. A simple, "We aren't in contact with him" should be enough information. Any further questions really are prying, albeit not necessarily with bad intentions. At that point, "I really don't care to discuss it" should end the subject.