WTBH -
My ABF says that stuff to me ALL THE TIME and so does his alcoholic mother. I have learned it is a manipulation ploy to get me to waiver and accept the drinking as part of our lives. It is soooo frustrating. They actually started to make me feel like I was crazy, a loner, and miserable. I mean, I am all those things NOW living with an alcoholic to some degree because it is depressing. But, ABF and living with him and my accepting it is what is making me nuts. LOL. I know I am a pretty mellow, reasonable, fun human being. I am starting to get ME back the more I work towards a future without ABF. I also get anxiety thinking about going through what you are now. No fun.
Just keep on keeping on. Soon (hopefully) he will tire of trying to push your buttons.