In saying "No" to him, maybe you will best be able to judge his response? If you dont want to spend the day with him then DON'T. It is about your boundaries now and his needing to respect that now.
Surely he also has a sponsor who he can turn to if he is so torn up? He needs to work through this.