He can blame you for all he's worth but you don't have to accept that blame. His inability to control his rage in front of *children* is a clear sign that you need to get him out again. It wasn't clear in your post if you had asked him to leave once more though...