You've said no to what? To your daughter spending time with her? I don't see where you can legitimately say HE can't spend time with her. And I don't know that you necessarily are entitled to complete veto power over his allowing your daughter to have contact with her. It sounds as if he is acutely aware of the problems, and as long as she isn't drunk or left alone with your daughter unsupervised, it doesn't seem to me that she would be endangered.