He's right about one thing. He's an adult and does not need to ask your permission to do what he wants.
Of COURSE you want him not to leave, but rather to make the changes you want him to make. But he has the same right to make choices that you do. You've made the (eminently reasonable) choice that you won't live with creeping addiction. Because that's what it is. I've never personally known any alcoholic who has successfully smoked a little pot and never picked up another drink. When you're an addict, one definitely leads to the other. It's still escape from reality, and when you're an addict, that's what you're chasing. So I can virtually guarantee it wouldn't remain an occasional toke or two.