When you have spent years and years tiptoeing around someone else's moods and temper, it can feel utterly impossible to take any risk of annoying them, let alone angering them. Too frightening, even if he isn't violent. You've been trained well. I know how that feels.
It's a really good idea to talk to a DV counselor about your situation so that you have some in-person support. Maybe just leaving a note is the best way, but they can help you decide that.
Sending you a hug.