To me, from what I have read just here, it doesn't seem as though you are using your AH. Instead, it sounds like he is trying to control you ....
For what it's worth, a common thread among narcissists is telling you that you're doing exactly what they're doing. Which is extra confusing and makes you take on blame and feel complicit. Individual counseling can help you tease out what's your responsibility and what's him being a nut.