I vote to do whatever is going to cause you the least long term stress.
It is possible that sending groceries this time is OK and then you make a clear boundary with yourself, and with him, that it will never happen again. His budget is his to worry about. Even if you share the money you can make it clear that you each have x dollars and after that - you stay out of each others financial/budget business (both asking about/mentioning and participating in)
I have children. I wouldn't want to worry about if they eat. It is one thing to hold an adult responsible for their choices, and another to do so for the kids. If he had no food would he tell you? You could stick to the boundary of not sending food with the kids but allow him to call you and cancel a visit if he does not have safe/healthy home for them. That would be his call to make, and his to own up to. It would suck for the kids but you would not be part of it.
Just some thoughts. I'm a million miles from having my life figured out so take it for what it is worth. Also - I certainly would not want to be in the position of policing visits. That would be awful and I'd look for a way around that for sure!