I can totally relate! My husband seems to think that as long as we have sex, there's nothing wrong with our marriage. When he does something stupid or says something cruel, my not wanting to have sex is completely blamed on me ("You've always been this way and you'll never change"). When he's unhappy, it's my fault and if I don't jump through hoops to try to improve it, then he's cold and distant.
I'm really trying to sit with that now, to believe that things are as they are and that I should focus on meeting my needs.