Therapists are sort of like dresses, sometimes you outgrow them, or they make your butt look big. Maybe you need to find someone new? Hopefully this akward exchange between you and he won't affect the client/patient relationship.
He could have handled the situation better. Also, therapists do get tired of clients. Sad but true. I know of therapists that are thinking of grocery lists and weekend plans while listening to clients. They can burn out. Hopefully he isn't burning out but something to be aware of.