(((HUGS)))
When it comes right down to it, you have to do what you are comfortable with and there is not one thing wrong with that. Life is not fair and there's a good chance he won't get what he deserves, but regardless, you have to look out for your own welfare and that of your children.
You don't owe anyone here any explanations. It is your life and you are entitled to live it the best way you know how.