It's all smoke and mirrors!
They say that love is blind. Your not alone and you are not stupid.
What he says is not truth.
Doing what you need to do for setting boudries and taking care of what needs be taken care of is not selfish and uncaring. Doing what is right takes more care and sitting about doing nothing at all would be more in line with not caring.
You are beating yourself up over the fact that you are a loving and caring person. Such a person is never stupid.