I was on (prescribed) benzos for close to a decade before I got sober -- they were also used at higher doses during my detox. Once I was done with detox me and my p-doc worked together to taper off of them in a safe manner. It took me ~ 4-6 months to completely wean off of them but it can be done. I still have anxiety issues which is why I was started on them in the first place (panic disorder), but the anxiety has greatly improved after removing the booze and finding ways to deal with my anxiety without having to medicate. My p-doc still prescribes me a very small amount of Ativan, with no refills, that we have agreed are to be used for "in case of emergency break glass" situations only. They are only to be used if I feel the onset of a panic attack occurring, which fortunately has yet to occur since I tapered off of them. Simply knowing they are there if things get really bad seems to keep things from getting really bad.
So basically what I'm saying is that as long as you are completely honest with your mental health care professionals I would trust them and not worry too much about what others think. I personally felt that I needed to at least try to get off of them. I had been on them for so long I didn't know if they were really helping or if I was just taking them to avoid the withdrawal symptoms. It turns out that I was able to function fine without them. BUT (big but), it can be just as dangerous to stop benzos cold turkey as it can to stop drinking cold turkey with many of the same possible symptoms. I would always recommend working with a qualified p-doc to set up a taper program for a safe detox from them.