Yep, human beings don't like change. People will have a strong, negative reaction sometimes. But, we can't completely prostrate ourselves just so someone else is not hurt.
I think you should say exactly what you mean, mean exactly what you say, but do not say it meanly or angrily. Easier said than done sometimes, I know.
I think you are doing great by upholding your personal boundaries!