If your son's uncle is a selfish, nasty person, whether he's high or sober, then you are more than justified in not wanting your toddler in his presence. Addicts genuinely seeking assistance are a different matter altogether.
Question is--who will watch him on Mondays? If you can make other child care arrangements, do so, unless your brother shows improvement (humility, gentleness, good humor).
My 21-year-old son is a recovering opiate/heroin addict, and I grew up with a severely alcoholic (and psychotic) father. Shield the children as much as you can.