Maybe cutting off contact with his family right now would be easier for your recovery?
I couldn't do it with my exabf's family constantly making me feel guilty; I know that they used to at times, but back then I wasn't ready to leave him.
I hope you can figure out some boundaries for yourself!
Hugs,
Savana