Instantly I get accused of "coercing", "controlling" and "dictating" and get told that it is indicative of an ABUSER (ha ha ha ha ha) to tell him that "unless he does X, I will do Y".
Anytime I made a suggestion to my X about how he might do/handle things differently, this is what he said. When DS12 was really little, he asked his dad to spend more time with him doing one-on-one things, and X said that he wasn't going to let a kid dictate his life. Doing things for other people is not just inconvenient, he literally doesn't understand why he would. "What's in it for meeeeeee?"
You set a boundary, and he's testing it. One of the ways he's testing it is the way he's made you break before, by accusing you of being controlling, crazy, and hyper-vigilant. You just know what he's doing now, you've named it and you've said you're not going to live like that anymore, and it makes you angry because this tactic is ridiculous, unfair, totally absurd, and completely unnecessary.
Unfortunately, you're just not dealing with a sane person. If only he were with it enough to know the same, yeah? Expect the crazy, respond with sanity.