Allow me to echo everyone's advice, here. RUN! I've recently left because of ALL the reasons you list here, including him being too drunk and not responsible enough to look after his own daughter. It WILL get worse. It DOES get worse. The anxiety can be crippling, trust me I know. But it is your body and mind telling you something is terribly wrong.
You deserve more. I never fully believed it when I was with my AXBF. But I believe it now. And it feels wonderful.
Wishing you courage, peace and serenity.