Hi and welcome to SR!
Please take Bookwyrm's suggestion and buy (and read) a copy of "Under the Influence" -- leave it laying around and I hope your husband will pick it up and read it too. It sounds as if you can both use a good education about alcohol use/abuse, and this is a great read.
To answer your question, "Should I worry?" Well, you could worry...but it won't change the outcome. I suggest you take that particular energy and save it for your child.
Glad you're here -- it shows some real maturity.