The legal aspects of visitation are sometimes very black and white--others not so. Yours sounds like it is NOT black and white. If you want to make sure--call up family court and verify what his rights are. It would probably be too much effort for him to try to file a contempt motion against you (he would lose anyway since he is not providing a safe environment). stbxah was not allowed overnights until he had adequate housing.
2 weekends ago older DS told me he was NOT going to his Dad's ever again because all he does is yell at him and say nasty stuff about me (and said he wanted to tell him to stop but thought he would start yelling at him again). My atty. told me to do everything in my power to get older DS to go with his Dad so I cajoled and eventually he did go. But after he left I realized I would never do it again because by doing it the message I gave to him is that the law does not care how poorly your father treats you (and neither do I), you are still required to go with him for visitation. My atty. told me if older DS would not go I could be held in contempt. Again, I thought--I am doing NOTHING to prevent older DS from going, he does not want to go. If it ever happens again I plan on doing. . .n-o-t-h-i-n-g. stbxah can call the cops and then he can explain that DS will not leave (or he will try to say that I am preventing it). It will be apparent that is not so.
Your daughter cannot talk and if the court gave you discretion over visitation and you are not abusing that discretion. . .then it would seem you are within your rights. You are keeping great records--keep doing it. If by some far cry he actually tried to file some frivolous contempt charge you will have your documentation--I would guess he will have none.