Well, in this case he's right, he didn't do anything to you, he simply did it. And you are correct that it affects you and everyone else in the household.
Just remember that while he has a choice whether to drink or not, you also have a choice whether to live with a drinker...or not.