Could you talk to a lawyer about it? Perhaps you could record the conversations and use them as evidence in court to get a different agreement worked out where he is not allowed to talk to her on the phone?
Could you explain to your daughter that he IS allowed to see her, that it simply requires a supervisor that isn't you and she should ask HIM to do so when he is allowed? I think whenever possible the truth is the best thing for children. I think a lot of times we don't give them enough credit for being able to understand it.
Above all else, just keep being the good parent you are.