I also distance myself from the family and feel bad about it too. But at the same time, I now recognize that it wasn't me who created this intimacy problem. I was trained from a young age, by my family, to not get too close to them. The relationship with my family is very damaged and has been for so long now and and that's just not my fault. So if I need to set limits with them and need space or have to process awful feelings about them or confusion or if I just don't want to associate with them at all at certain times, well that's just a natural progression of this damage.
I still feel bad about it so I'm not sure what to suggest about that AP, but I find it helps to just look at the facts. I didn't create this. I empathize with your situation.