Benzo withdrawal is hard but I don't think one pill in three weeks is going to affect recovery. You doctor doesn't think so either.
It's hard for doctors to really appreciate benzo withdrawal. How long you were on it makes a big difference. Also, while acute alcohol withdrawal lasts a matter of days, it was months before I had any semblance of physical normality.