There are two issues I think. One is people's perceptions, the other is people's actions. The first is important to let go of, as you know. I work on that a lot myself (especially by reminding myself that I usually don't actually know what, if anything, people actually think of me!).
But in your case people are taking the action of sharing false information at your workplace, which is an actual harmful thing to do. I would take steps to calmly correct the information in whatever the most appropriate way is. Maybe pulling aside the person or people. It seems like rumors like that could hurt your employment.