I think it's perfectly understandable that you felt those feelings, but think about it--how would YOU feel if he said, "Well, there were times when you were so cold to me I wished you were dead." Now, that might have been TRUE, but expressing a wish that someone comes to harm is a whole lot different from simply stating that you were angry and helpless (which is what it comes down to--the "wishing he'd fall" was really just part of that--a part that didn't need to be said).
But, as noted above, what was said was said. If he wants to use it as an excuse to drink, he can--an alcoholic can use ANYTHING as an excuse to drink. Or he might just need some space after being hurt.
Have you apologized for saying something that hurt him? You don't have to apologize for how you FELT--that is not within your control. But for the fact that you said something that hurt him.
I agree that Al-Anon would be very helpful in learning to keep your focus on your own actions.