I let mine in a time or 2 as well. He knew all the right things to say and do to push just the right vulnerable buttons for me. And you know, it was a BAD thing each time. I remember the last time, after the police took him away. I went to the bathroom mirror, looked at myself and said "OK, who in there opened that door???"
My counselor told me that was a good thing to ask: which part of myself was still so small, afraid, wounded etc that it would make a decision to let an unhealthy and dangerous person into my home. She recommended some good strong self-talk so that my stronger, healthier thoughts would have a louder voice. She also suggested putting yellow sticky notes by the places that were troublesome for me - in my case that was near the computer, near the phone and near the front door. I wrote notes to myself so I'd remember what to say when he called/emailed/texted or came by. Which he did, in violation of the protective order - because, like so many others, the rules just didn't apply to him.
I understand completely, and I strongly recommend you take steps to protect yourself. The man you love might be in there, somewhere, but until he gets help and LOTS of it, he's dangerous to you.