He'd often claim that he only drinks when I'm around.
This jumped out at me. Not only is it a bald-faced lie, it is him blaming you for his drinking. This WILL get worse and if you allow it, you will eventually start to believe it. Please! If you feel divided now, you will feel even more so after years of being blamed for things.
All we can do is share our experiences with your situation. You have received many such stories here. I hope you will follow your head, because I believe deep down, you know we speak the truth. Your situation is NOT unique. Your addict is NOT unique. His words and actions are straight out of the alcoholic manual.