Whether the power is outside or inside of me never mattered. Power is invisible, and open to interpretation.
I agree. People who get upset when they hear words like higher power are more or less less expecting everyone else to conform to their narrow-minded viewpoint.
"All expectations are seeds for resentment".
(the original Laozi old man)