One thing that I have to remember every time I start feeling guilty is that I love my daughter more than she loves me.
Not to say she doesn't love me...it's just the dynamics of a parent/child relationship. They KNOW we care and love them with every fiber in our being - and they tend to exploit that - because they can. Making us feel guilty is easy. Counting on us to come running and save the world - almost guaranteed.
Think back to your own parents. Yes, we love them...but our universe does not revolve around them and our heart doesn't break with every wrong choice they make. Concerned and worried? Yes. Absolutely heartbroken? Probably not. Our world wouldn't come to an end because they denied us something we want. We still love them... life goes on.
In other words: Our saying no is inconvenient and unwelcome, but very rarely will it ruin their life or destroy their affection for us. Saying no hurts our hearts more than theirs.
I never want my daughter to feel abandoned, either...and I still answer her calls, even though I cringe and wonder "what now?". It's VERY hard, but I'm trying to learn to be an ear and listen, not a rescuer, unless her life were in danger at that very moment. (And I think we can all tell when there is a genuine crisis and not just another darn-I-am-in-jail-again.)
It's hard to say no, and to think they might not like it or might feel disappointed, but they will not hate us for setting boundaries. When they were babies/toddlers/teenagers they accepted those boundaries and griped or threw fits, but accepted them, still.
As adults it's the Same game-Different Playground.