Welcome to Sober Recovery.
Even if you aren't drinking "alcoholically" the amount you are drinking will take it's toll on you physically. Doctors recommend no more that two drinks a day for men. You are exceeding that. So for health reasons alone, you should probably cut back.
If you can't cut back, or moderate your drinking, then you do have a problem.
I spent my whole life trying to control my drinking. I found quitting easier.