I find accusing such a strong word, he throws it at me all the time, I see it as merely stating facts, or calling him out on his bad behaviour
I think this is a good example. In fact you are accusing him - well that is the way he sees it and since he is the recipient, you are going to be hard pressed to convince him otherwise.
If you look at it the other way around, you think you are stating facts, he would be pretty hard pressed to convince you otherwise?
This isn't something you need to get wrapped up in if you want to truly detach anyway, you have made your boundary now.
As SK mentioned, I also don't think that detaching is a long term solution but for now it just might cut out some of the "insane".
Ok, maybe it is better not to talk at all
This is actually true. Detaching is about letting all this stuff go. Just let him be him, you be you, you don't need to call him on anything. In fact how has that been working? Not at all I'm guessing. It can't hurt to try a different approach and maybe it will give you some peace of mind.
It's not easy but it can be done.