I'm a lifelong atheist and while I struggled with this concept awhile back, I no longer do. To me, "Higher Power" is shorthand for a handful of things that basically take me out of my body and anxiety and help me give up the illusion of control.
Things that do this are things like being outside in a natural environment ("the world is bigger than me and my problems"); learning/trying/achieving new things, especially for whatever reason physical things, like taking on biking this year despite being a single mom with two kids and two jobs ("I am capable of change and growth"); learning to meditate, and despite being very anti-woo-woo this has been a big component of my emotional growth ("I am deserving of self-care, love and peace").
There are a variety of other things, but when people say "higher power" it to me essentially means the confidence I have that I will survive, thrive, and grow. A friend of mine, also an atheist, is always on the lookout for "magic" -- things that are beautiful in their ugliness, like the filament on a broken lightbulb, an arrangement of sticks and wildflowers. She's an artist and you wouldn't believe the stuff she comes up with. That ability to see and create beauty out of trash is her higher power -- and quite a metaphor for personal growth.
I don't know if this is the case for everyone, but through my youth and through the darkest points in my adult life, my atheism was tied directly to a sense of nihilism, or a case of the **** its, or even the **** yous. Not that I would believe in god if I'd had an easier life -- I don't think that's the case. But the sense that I wore my ambivalence or hostility to religion on my sleeve was tied to my experiences of trauma and resultant emotional hardness, and as I've made progress healing from the trauma I find that I'm less bothered by representations of religion in my life and it's easier for me to roll with it than be immediately turned off by it. My ability to "take what works and leave the rest" has been monumental in this process.
As far as spirituality, again, I'm an atheist. But when I'm in the middle of nowhere on my bike, with my beautiful children with me, spotting birds, deer, bees, squirrels, a nice breeze, the sun on my skin, the wind in my hair, I feel so much abundance in my life. That feeling of being grateful to be alive -- that's my higher power. I live for that.