Daily contact is a bit much IMO. No one (myself included) has a custody agreement that outlines daily visitation. My exah demanded something similar (daily visits to my home to see our son who was then 13) and I said no. No judge is going to give him that and you don't have to either. Daily short visits are disruptive (to you and the kids). Why can't he take them for longer periods less often? If he is unwilling to care for them without you present that is very telling (he is more interested in seeing you than them).
IMO a visit to an attorney to get temporary orders in place (that do NOT include you and he being present at the same time) is needed.