The fact that he told you that he was thinking that he wanted to hurt you, combined with the fact that you know he used physical violence against someone else for reasons other than self defense, means that he is willing to cross a line that many will not.
Violent people get violent.
You need to think about that and decide if thats acceptable in your life.
If he's drinking, it is unchecked, and unpredictable.