Personally? I would call a meeting with all of your kids and explain to them how it is.
I know this scenario, the blaming, the paranoia, how people are ripping him off blah blah blah, quack quack quack.
Growing up (well as I got to be older) I knew to take everything said with a grain of salt, so it was a given after a while. I don't know how old your kids are but they do need to be put straight about this. Somehow you have become the person in question here and that needs to be nipped as soon as possible.
It should also be clear that you aren't going to JADE - Justify, Argue, Defend or Explain every little situation that comes up.
I hope he didn't believe his Dad?