I agree with you that us alcoholics usually have low self-esteem. If I had cared about myself, I wouldn't have continued to poison my body.
Early sobriety is a time when you may need to make some big changes in your life. Abuse, of any kind, is completely unacceptable. If this man is a trigger for you to drink, then what can you do to change that? Recovering is hard work and you might need to make some changes in your life to get past Day 6. But, know that you can do it!