Liz, is it possible that the words he chose touched a nerve?
If so, just know that is is MORE THAN OKAY for you to have a night off from being Mom. If you want to go out on a date then you are allowed to do that without having to worry about it being perceived as fobbing your son off. This is your ex's problem, not yours.
I agree with the others that the person you need to address this with is your son. You can't make STBXAH follow the parenting plan or be a decent human being, but you can make sure your son knows that he is a priority to you and nothing anyone says can make that not so.