Have you read the Big Book (pgs. 72-75) and the 12 & 12 about the 5th step? It's very clear you can do it with anyone. Many people in AA will tell you otherwise.
I had this happen to me last year. A sponsor was pressuring me to do it with him. I ended up not doing it with him. I'm no expert, but my only suggestion is finding someone you're comfortable with.