Stick to your guns and stop debating about it with him. Don't listen to his tirades, just hang up the phone. Be consistent and they will eventually stop. You are going to have to be the one to put a stop to this. Do you have any kind of legal agreement regarding visitation, child support, etc. Until you do, you don't have any backup.