So the "visitation" is just something you decided out of the kindness of your heart? What is he taking you to court for anyway..
Move. Better yourself and those children.
Due to the abuse he likes to offer, it might be best if the courts DID oversee his supervised visitation, leaving you out of it.
Learn about your rights, you'll need this knowledge if he's going to play like this.
And I agree.. I have never heard of 3x a week for 4 hours!?! That is a lot of your time.