True, he is an adult and can choose whether or not he wants to smoke "a little" pot. But you have every right to be upset about that and not like it.
One thing to keep in mind: Addicts tend to minimize their use...."I only had a few beers with the guys after work", "It's just a little pot, no big deal.." "I only use once in a while, not every day", "It was just a few beers, that's all; no hard liquor.....". Same with someone who cheats on you. "We're just friends"....oh, really? Hmmmm. That's not what she's saying....
Ok, you get the drift. If they tell it's just a bit or a little, you can safely assume it's more than what they say and will continue to grow into even more. Addiction is progressive.