He's ashamed of himself. That's a big deal. And an even bigger deal to admit. His feelings are his, he has a right to them. You know why he's quiet. He told you it's not directed at you. So . . . ? Isn't it good he's admitting to that?
We can't go through life taking other people's feelings too personally. That's puts us on an impossible rollercoaster. And yes, two languages. Man / Woman. Addict / Not and in some cases /Codependent. There are different languages here. Men aren't hairy woman!
We do need to learn to talk to men and they need to talk to us. The goal is to start meeting in the middle.
Have you thought about marriage therapy?