If you are too scared to speak up, he's created that fear in you in subtle or not-so subtle ways.
I have been there. I remember opening the pantry door, finding ah drinking in there, shutting the door and going about dinner with the family as though nothing had happened. That's how I lived for years and years, though that may have been one of the rare times I actually caught him in the act.
When I finally started Al-Anon, i found that I still hardly had the will to speak to him about anything. All my energy went to wanting to get him out of my life. I don't speak to him. The drama beneath it all was practically word-less. Every word I did say in that regard led to a big fight and a few times, police intervention.
The trouble is, when you stop saying anything and go about your day in spite of the drinking, do the words in your head just fade away, too? When you stop thinking for yourself because it leads to too much strife, then you have a whole other set of problems. You start losing yourself. And that sets the stage for more abuse towards you.
The sooner you reclaim yourself and your life, the less psychological damage you will have to try to undo later.