So I talked it over with my sponsor and this is what we decided would work best.
Instead of reacting, I took time to figure out a response. I texted him and told him he can call at 10am tomorrow to talk to the kids. I also laid out my boundaries: That I expect him to be sober, have a nice short conversation with the kids, and if the conversation was at all inappropriate, I would end it.
So we will see if he bothers to call. He usually doesn't. But either way I am holding true to my self and my boundaries.
