Been there, done that, have the attorney receipts to prove it.
Selfish? Hardly. But if that's what you want to call it, so be it. Self-preservation would be a better way to put it, though.
I felt guilty too. I stuck around a lot longer than I should have. The guilt passes... hang in there.
D