Your an adult, you are entitled to go out and have some fun. He is not your father, and you are not a child.
This is all about control, most addicts/acoholics have serious control issues, they have no self control when it's about them, but, want to control everyone else...in an attempt to keep others in a box.
I wouldn't give his words a second thought.