My mother had married 4 different men by the time I was 10 years old so I had 4 fathers. None of them were alcoholics but most of them were abusive anyway. When I was young I was angry with my mother for making me go through those relationships with her... but then I grew up enough to recognize that my mother made poor decisions regarding men but she was a loving mother who did her best for me and the men were the bad guys. Sadly, the father that hurt me the most was the one who wasn't abusive... he was the only one I cared about but he left. To this day, despite the amount of abuse I received from other fathers, the loss of the one who didn't abuse me hurts the most.
What I'm trying to say is... we are all hurt by things that happen to us in our childhoods. No one can say what is the most damaging thing that can happen to a child. It's different in every situation.
I don't know anyone who had a perfect childhood. All we can do is give our children the best childhoods we can given our situations and abilities, all we can do is our best.