There are people in this world who will blame others for all of the bad things they do in life.
Rapists will blame the victims, their attire, their actions.
Pedophiles will blame "their disease"
Even murderers will find a reason why they simply couldn't avoid taking another's life.
Would you feel guilty with them too? If they attacked your daughter, molested your son, killed someone you care about in anger? At what point will you let go of the guilt, BeginAgain? At what point does he get a wake-up call that his behavior isn't going to be accepted if he wants to see you & others at his doorstep?
What does he have to do before you'll realize that you are the GoodGuy (well, goodgal) here, and he is damaging you?
I really, really do understand the guilt, but working through it and getting rid of it is a fight that's worth fighting. The freedom it can gain you is so life-changing. I've been there too, and the difference in my life is like black and white.
Hugs and strength to you
GiveLove