I think you should do whatever will feel right to you, without being concerned about what he may or may not say about it. They will do or say whatever they want to no matter what we do or don't do.
Your lack of response may give him something to say as well, but it doesn't matter. You matter. Many of my exabf's friends think I am horrible for not taking him back. I have to not concern myself with any of that. I know what I know, and what is best for me right now, and I find strength in that.
Do whatever you need to do to have peace in your life.
