gneiss, don't be a victim of others' opinions of you. Their opinion of you should not determine your opinion of you. If they choose to see you as the black sheep, let them. Remember nothing others do or say is because of you. What other people do or say is a projection of their own reality.
LaF's suggestion to establish boundaries is excellent. Personally, however, I have the application a little tricky. Especially around people that you have already granted extraordinary permissions.