I think the question is...could you live with yourself if he seriously hurts one of your younger children (I know you know the answer to that, because you've already made the right choice). He's already seriously hurt you, and has already assaulted your 10-year-old. I know it's hard to kick him out, and that feeling of hurt is now occupying your heart and mind...but it is far easier to live with that course of action, than it would be if you let him stay and he seriously injures your other children. That would be far more difficult to live with.
You don't really have a choice. You have a greater responsibility to your younger children. He's an adult. He has chosen this consequence for himself.