My advice would be to stick like glue to the court-ordered agreement. Simply because it doesn't sound like your ex has a lot of respect for either you or your daughter. He's clearly not seeing your daughter for HER benefit, but for his own. And he's using her as an emotional garbage can which makes me (like nwgrits) unreasonable angry. Been there, done that, still dealing with the fallout in my kids.
He is not doing supervised visitations because he can't manipulate your girl under those conditions. Phone calls is perfect for a manipulator because there is no record, there is no evidence of what he's doing.
Is it possible to go back to court and say that you believe he is being emotionally abusive to her and request permission to tape the conversations? He will no doubt oppose that, but I bet he would not call --or want to call more often-- if he knew he was being taped.
Again - him calling her twice a day is not helpful to her in any way, given his state of mind. I would want to limit that contact - and I think you might be able to if you bring up the fact that he can't be bothered to do ftf visitation with a supervisor...
But I'm not a lawyer and I'm just thinking with my gut here. That poor girl should not have to deal with an adult behaving like a child.