Next time, try to remember that "No" is a complete answer. You don't owe him anymore explanations. He's just doing what addicts do - pushing boundaries. And he's getting a response that he didn't expect - your standing up for yourself. He doesn't like it so he's trying to manipulate and make you feel bad.
Like I tell my 5 year when he tries to argue with me about my boundaries when it comes to his behavior, "I love you too much to argue. I'm taking a time out now. Feel free to go to another room in the house." And when he pushes, I just repeat it over and over and over until he just walks away in frustration.
You can please some of the people some of the time, none of the people all of the time but the only way you can please an addict, is when you give them what they want.