I am/was a really bad binge drinker. Like you for the most part it was 1-3 beers at night with dinner or watching TV, but then I would rage at least 3 times a week if not more. Then those binges turned horrible and I was constantly in trouble when I drank I would be crazy, fighting and getting myself into some awful situations and those binge drinking sessions I had no control over. If I start I can't stop.
I really didn't think it was a problem because it's easy to minimize binge drinking because it's not an every day thing. You're not the picture of an alcoholic. But if it's effecting you outside of that night then it is a problem and worth taking a look at and trying to quit.