If he literally will not let your child go until you relent then that is kidnapping once anyone (or anyone in your care) has declared that they are leaving and he stops them that is holding them against their will, in this situation you need to take out your phone and call 911, tell them where you are (preferably as Soaring Spirits advised in a very public place) and have them come and make sure he releases your child without conversation, and then I would file for sole custody with either no visitation or supervised visitation.
My x-wife used to stand in front of the door or throw her self down when I tried to leave, the police finally had to tell her that this is a felony, even if no one is threatened or harmed.