I agree, just "no".
Explaining can open a door for argument and weaken your position. You don't need to explain; he'll figure it out.
I understand how difficult it is, but it helped me to remember that a lot of the time I wanted to help so I would feel better about the situation.