Evidently he has a right to an investigation before his parenting time is changed. If the investigation is determined to be "founded" then the court will modify his parental rights/time.
The least they could do would be making it supervised.
And
YOU HAVE NOT FAILED YOUR KIDS. You are feeling that way because you know how valuable they are and you want to protect them. And you are. You are doing everything in your might within legal boundaries -- and that is what you need to do to protect them long-term.
If you did something drastic, like took the kids and ran -- which I'm sure you've thought about; I know I did -- you would make things worse. It is horrible that you can't do anything about the visitations. I hate this for you.
And I agree with you that the protection the law & authorities gives children is abysmal. Nothing short of abysmal.
Lots of hugs. Lots.