You would not be letting him down. You would be letting yourself down if you did not live the best life that you could. With him or without him.
And it is not up to you what kind of life he has. He seems to be making that choice just fine all by himself.
Hugs,
JT