Well, it seems to me that they are within their rights to ask either one of you for whatever they want. You made it clear all along that you couldn't afford it, their dad had hinted that he would, and they needed to find out what his decision was.
It's too bad he was such a jerk (thank goodness your kids are getting counseling), but I don't see that you did anything wrong. It's not as if you said, "Ask your deadbeat dad"--he set up this situation, so he's got nothing to complain about.
I think you're reading it right--he's trying to get a rise out of you. Don't take the bait. You can't have a reasonable discussion with him about it. Rant about it here, document it for court, and hug the kiddos. You can share their disappointment without making it a bigger deal than it has to be. Lots of people can't afford to do everything they'd like to.
Hugs,