How about just saying, "Thanks for your input, I'm fine with what I'm wearing." And that's it. It's really not necessary to label it as "being a jerk"--that's what escalates the situation.
Your choice of what to wear to work isn't up for discussion. Just treat it like the irrelevant comment that it is.
Is he normally that controlling toward you? If so, that can be a red flag for abuse. If it's just him being grumpy because he's gone six days without a drink it isn't worth arguing about.