You can do your own research on the origin of popular sayings, too.
It’s Up to You; A Story of Domestic Bliss, by George V Hobart, dated 1902: “I see-sawed back and forth between Clara J. and the smoke-holder like a man who has lost some of his marbles.”
Random House Dictionary of American Slang