Queenie,
Here's something I learned that has gone and continues to go a VERY long way in my Recovery:
Given that it seems you feel bad for your physical behavior towards this person, your personal values (and morals) probably include "I do not wish to harm other people."
When you act in opposition to your personal values, no matter what the situation, you negatively affect your own self-esteem.
I have to remind myself of my own values and morals at least every week.