No, you can quit anytime you choose to. Lots of folks quit drinking and still have their families, marriages, jobs, most of their health. They quit drinking and got on with their lives. There are others who don't do that, and they get most of the press.
I think you said a mouthful when you said you pick up drinking again after a period of sobriety 'casually, without much thought'.
The answer for me was to become aware of this, to become mindful. This stopped the autopilot phenomenon that we read about, 'I don't know how it happened, I just got plastered again'. I started becoming aware and mindful of thoughts too, and this is what kept me sober. Check out AVRT if this sounds interesting to you.
When you think about it, if you do, it is pretty silly to think that we can obtain alcohol, pay for it, put it in our mouths and swallow it, all without sufficient awareness to stop if we have made a solemn decision to quit. If we haven't made that solemn decision and say instead 'I'm going to give this sobriety thing a try and see how it goes', it's a guarantee that the future is not going to change.
Jack, you can quit drinking (and you will) once you decide that your life will be better when you do, and that you deserve to have this better life, the one without alcohol. Are you ready to make your plan about continuing to use alcohol?