I suspect people tiptoe around him because he's done a good job of manipulating people into feeling sorry for him.
It's hard for me to imagine NO ONE he's ever run across has pointed out that he's a bum and a mooch. Maybe nobody in your family has, but it's hard to imagine that he's reached the age of 50 and nobody's ever suggested it to him. I suspect, though, any such unpleasant thoughts go in one ear and out the other--anything from you would probably be treated the same way.
I'm of the personal opinion that the fewer the words, and the more quietly said, the less harm to anyone involved. A diatribe from you might just add fuel to his own resentments (though I know it's hard to imagine he has anything to be resentful about, he probably has them).
So probably the least damaging thing you could say (least damaging for all concerned) is probably just to politely, but cooly, decline and ask that he not ask again.