I can hear what you're saying, but as others on this board are so fond of saying "you need to play the tape all the way through". You didn't just stop eating dinner with him on a whim. It's your boundary. AH drinks = I don't cook/eat with you. Is he still drinking? Yep. Then don't feel guilty about being away. He doesn't feel guilty for drinking does he? If you need to say anything to him, I wouldn't apologize, I would just restate my boundary. You're more than willing to eat dinner with him IF he doesn't drink. KWIM?
Edited to add: His comment about you never being around sure sounds like a pity party to me. If he wants you around, then his actions will show it. Don't get sucked into his pity. You've grown too much for that. (((HUGS)))