It really doesn't matter what he thinks about how you've chosen to exchange your daughter -- it's YOUR choice and YOU get to choose! I think he's just mad because he doesn't get to see the sad look in your eyes, loses the opportunity to vilify and abuse you, and feed his own sick ego. I think he is eaten alive with guilt and is trying to find an inroad for justifying his actions. When you don't engage, he's got nowhere to look but in the mirror...and that is making him very uncomfortable. PLUS....he's lost his "dance" partner. Oh yeah, he's really got a mess on his hands now.
Did I read somewhere that he left with your daughter? So he's got her 'full time' and you don't?