My ex would say the same thing then if I did stay after school for a game or go out for dinner with a few co-workers, he'd call and say when are you coming home?
He'd use it against me for always being away from the house. After awhile, I didn't want to go home because I never knew what I was going home to. I guess I was just supposed to sit there and read my books quietly while he drank.
No thanks! Not a relationship that I want again.