It is because at one point your ex was attainable/obtainable but now your ex is not. It is part of your life. And that is what you are holding on to.
When someone talks about your ex, it is almost like hitting your self confidence of who you have chosen to be with in the past. And even worse, your ex does not want you.
I will say this about self esteem--you have the choice to be with someone who has positive traits--they smell good, they have good hygiene, they brush their teeth, they comb their hair, they are loyal, they are kind, they are financially responsible, etc. And it starts with yourself to have those positive traits as well.
I learned a lot about myself by dating different guys before I met my husband. I knew what I really wanted in a relationship and who I really needed to be to get to that point. It was a learning experience.