You "feel unsure" because in the MOMENT, no doubt, it felt good--familiar, nice.
As someone else pointed out, it's exactly the way an alcoholic feels after that first drink.
And equally dangerous.
Don't misinterpret your "gee, that felt good" as some kind of indicator you should consider taking him back. One has nothing to do with the other.