Can he visit your son at your home with you there? You are right to protect your son and "no" is a complete sentance. Make sure he cant check him out of the daycare without your permission and hang in there for the remaining two weeks. Maybe consider going no contact for the next two weeks.
Good luck, stay firm on your boundary -