Hi Mama, of course his relapse has affected your relationship. Most men seem to think that intimacy makes things better, whereas for women things have to get better before they feel like becoming intimate.
His anger won't be helping the situation. If you go into intimacy feeling forced or coerced it won't be enjoyable for you, and might make the situation worse.
The fact that he tells you your work on yourself 'shouldn't matter' isn't a good sign. He's dismissing what's important to you and your recovery because he wants his own needs met.
Frankly he seems to be putting his needs ahead of yours all the way and falling back on 'I'm your husband', meaning he feels he has rights over your body. If you get to the point where you dread physical contact, the future for your marriage isn't good, and you may want to think about a permanent separation.
I speak from personal experience because in my own marriage I realised it wasn't viable when I knew I would never enjoy sex with my husband again.