"I'm pretty involved with a group at church, and I like to read a lot." -- generally my response except I add in the easy excuse of kids' soccer games and the timesuckers my boys are :-)
My closest friends at work know, no one else does. Like it or not, I work in a very judgmental profession.