Well, his addictions and behavior should bother you. That's okay. That's normal. Don't beat yourself up about it. They bother me and I don't even know him.
Here's one thing that is helping me immensely, dealing with my own AH (married 16 years; 3 children, like you). Have you tried meditating? There is a free Oprah/Deepak meditation series coming up soon, online. It takes 10 minutes a day. I did a series in the fall (actually only 5 of the 21 sessions) and found it really helpful in disentangling myself from the craziness of my own mind. Now I'm taking a meditation class and continue to find it the best therapy ever.