Nothing that bothers you is unimportant because YOU matter! We all have to learn that lesson around here! ;-)
I don't think it is enabling at all for you to take care of your own sweaters. I would think that is more like a boundary that you are setting, and that is a good thing.
Just keep your sweaters seperate and wash them yourself if that makes you happy!
(((HUGS))))