In my opinion, this isn't about fair. This is about acting in your son's best interest.
Court-ordered visitation pretty well spells it out in black and white for a bullying alcoholic.
Kudos to you for letting your son continue to enjoy his day with his cousins!