Sounds to me like you are way over-thinking things, which is common for most of us. It's just as likely that he's busy and hasn't gotten back to you yet.
You deal with it by just going on with your day and doing what you need to do to stay sober and do what is best for you. You made a decision that was most likely the best for you by not attending the party, and that's a perfectly acceptable solution. Perhaps at some point you can have a conversation about why you didn't come - a true friend will certainly understand.
But bottom line, don't sweat it - you cannnot control what other people think and you'll drive yourself nuts trying ;-)