I hope for your own safety, and that of your child, you will keep the protective order in place.
You are not being cruel, no matter how many times he tells you that. He could tell you that the sky is green with pink polka dots, but that does not make it true, right?
Sending hugs!