Good for you for questioning your motivation. That's important stuff.
What you said here caught my attention:
It would be my 'little secret' that I could think about if she starts getting in my face.
Fast forward to later. I think you have to decide for yourself how having a 'little secret' would make you feel about
yourself. IMHO, having a 'little secret' fits in the category of infidelity.
And one other thought: Your ex-girlfriend is married, right? With respect for her husband--would you invite her husband to coffee too? Or are you looking for an exclusive meeting? Please consider his place in this too.
I have a feeling there are healthier ways to remind yourself of who you once were. With the help of some old guy friends, perhaps?
