I hate letting people down, too..... but you post reminded me of a something I read somewhere that really struck a chord with me at the time:
Betrayal of oneself in order not to betray another is betrayal nonetheless.
If they care about you, they'd want you to do the best thing for yourself.