Yuck... I am sorry you have to deal with that. Your story gave me a snapshot of what my life would've been like had my alcoholic ex chose to stick around rather than take off to Party-Land. Single parenting was hard but I'd take it any day over dealing with an unreasonable A trying to badly co-parent. It is nice when they just go away.
Anyway, I think you did the right thing. I have zero sympathy for anyone who only has their own schedule to manage trying to make someone with small kids work around them. You have 2 little lives that come way before his and if he was "Super Dad" as he wants to appear to be, he'd be more concerned about the little one's health and getting to the doctor than finding a bone to pick with you.