Agree with wren - I'm sure I've done damage to myself, even if I don't have any serious problem right now that's obvious. Probably most of us will never know the extent of the damage - it might be at the cellular level, ya know?
I understand your worries, though, because I had some soreness in my liver region for the last several months of drinking. Those worries definitely gave me the desire to stop altogether.