When I hear, well what did you expect, it makes me feel like I should let go of my anger and sort of be a doormat again.
No, it doesn't mean that. Your anger is completely normal and you don't seem to have unrealistic expectations. You're taking the right actions and bravely taking the necessary steps to sever the ties. Just focus on what you're doing right and try to let go of negative judgements about yourself.