My fear of my children's reaction was what kept me from taking action for a long time.
What made me change my views was was when I realized he was not acting in a fatherly way. They weren't really getting much of a father. He was always passed out or too intoxicated to interact.
Learn as much as you can about your options. Your happiness is also important.