I can't say why you feel guilty, but I was ashamed of myself and the life I was leading.
I guess the easiest way to get rid of the immediate guilt (and this isn't meant as a wiseguy answer) is to...not drink and get yourself into situations you feel guilty for.
Find the support that works for you - make the changes you need to make.
As I stayed sober and sober longer I had more positive memories to replace the bad ones - I did my best to make my recovery a kind of living amends for the things I'd done wrong before.
The past is done - we can't change it - but we can very much affect our present, and through that our future.
Start again Critter. Start today
D