I was listening to a "spiritual" speaker the other day.
He said that almost all of the greatest people in history were "free of the good opinion of others." In other words, they did what was right, it just didn't matter what anyone else thought.
And you are doing what is right.