I had a very similar experience recently. I had run into my AH on the street and after a conversation he leaned in to kiss me goodbye. I had decided that the time previous to that he had hugged me and it threw me for a loop. So I took a step back. He looked hurt and I felt a wee bit guilty-but also proud that I was doing something that put my own well being first. I figured we don't live together so I did not have to deal with the fall out. I almost emailed him to explain but then thought I really did not owe him an explanation.
Fast forward 2 weeks later and we were talking on the phone. He brought up the incident and I explained why I did it and I said it was difficult for me but I needed to take care of me first. He seemed to understand. I also told him I was willing to take it one incident at a time and I would appreciate it if he would respect those boundaries and I would respect his-boundaries are very big for him right now.
A month later at his father's memorial service, he came up to me to say goodbye. He had this look on his face like it was painful to interact with me. I hugged him and said a few kind words and ended with "I Love You". He backed up and said, "And that's why you backed up when I tried to give you a friendly little peck on the cheek that day?"
I reminded him that we had spoken about it and that I did not want to go over it again at this time but if he wanted to have a conversation about it another time I would be happy to.
He said he did not remember talking about it!
That was scary for me to hear him say. I don't think he would lie about it like that.
and mind you- this is the man who has said that I make his skin crawl when I touch him and that he has no interest in kissing me because I am a bully.
I am not sure it is pure manipulation. I may be naive but he does not play head games like that.
I do think his self esteem is so low that he cannot handle any rejection.
I think he is conflicted in how he feels about me.
Again, I remember an AA speaker saying once
"Treat me special and I feel normal, Treat me normal and I feel rejected."
I too have to pick and choose my words very carefully when I talk to him. Its a lot of work. I don't want to live my life like that.