I think you may want to think long and hard about this situation. Your "full codie mode" is hurting your child...and I give the ABF very good odds that he will continue to go bac and forth because he has so much power over you.
I hope your son finds a happy home with his father, love, ATTENTION and stability to help him grow up to be a strong good person. He knows right from wrong. ...You did not consider his feelings when you went through the cycle of taking ABF back and you broke your word.
who can blame him that he does not trust you?