Two things I thought about in this situation--
First, his nature likely makes it hard to feel totally "safe" from him. Like he may try to learn and use little details about you to use against you in some way. It took me a while to realize he couldn't harm me anymore, even through the kids--because they started to see him too. And even friends think they're being helpful sometimes by sharing seemingly harmless details. I understand this fear. It's fine, but eventually it passes.
Second, and I struggled with this one more. If someone is truly okay with him and his behavior, KNOWING what transpired, I simply can't be friends with them because we don't share the same values. Not punishment, just choosing who gets to be in my space. And that's okay.
You get to choose your friends!