A parent's job is not to be a "good guy".
Its to give love, support and protection to thier child.
Its to teach them how to be responsible, kind and self-respecting.
Its to make decisions that people wont like or undertstand in order to accompish the above.
Your powerless over his fathers stupid words to him, but your not powerless in how to protect him from his dumb decisions.
Do you want to protect and care for your child or do you want your child to "like" you?
The ex's of our lives do tend to put us in that position of being the bad guy. So be it.
Make the decision thats best and healthiest for your boy, explain to him you have a difference of opinion than daddy's, and its ok for people to disgree, and its also ok if you r boy doesnt like your decision.
Id rather have an angry child at me, than a dead or injured one.
If the ex refuses to enforce your agreement for child care, consult an attorney.
Good luck!