He can say whatever he wants. The fact is he has specific days and times to see the children. If he doesn't adhere to those well so be it..his loss.
Put your foot down starting now. I imagine that once he's see's you mean business his shenanigans will stop. You didn't mention how old your daughter is. Your children don't need to know why going on behind the scenes. That's adult play. Believe me they will know well enough when they are older. Do yourself a favor and stop including him to parties ( I'm sure that these kids must get a knot in their stomachs just hoping he won't act crazy anyway). Not cool. If he wants to do something special for them on their birthdays let him do it on HIS OWN TIME.
Good luck girl! Stay strong!!