No, it's not cruel. You're putting yourself first, and for the right reasons. You've tried visiting before, and it sounds like you left unhappy. If nothing's changed, you will likely leave unhappy again. I think it's always okay to remind the addict that you will always be there if they choose recovery; you don't have to visit him to tell him that.
Hugs.