Spooner, you are worrying about him, how he feels, that you are being callous. Who left for 5 days? Who is really being callous here? You say yourself his behaviour was horrible towards you.
My experience is that the awful feelings are fully our own; the fear of finally taking baby steps to confront the terrible things we put up with. Although logically it shouldn't be, it's scary. Standing up for ourselves is not as easy as standing up for others - we feel selfish, insensitive. Now that I'm in my own recovery, it surprises me how much we can drag ourselves through the dirt on behalf of others. Why are we less important to take care of???
You did the right thing if you feel that it feels right. Only you can make that decision. But along with the horrible feelings, if you feel any inkling of peace, then you know this is your path.