Um, no. As a mother, I know exactly how you feel. The only people who have watched DD have been close family members I know I can trust to uphold the parenting choices I make AND care for DD's particular needs. Oyah and once I had her babysat by her daycare caregiver, whom she saw every day and DD FREAKED out so badly I never repeated the experience.
Considering the fact that you know all about addiction, I'd say your instincts are bang on AND that your RAH was perhaps trying to bait you into an argument, since he knows full well how you feel about childcare and a potentially intoxicated caregiver, be it your RAH or some stranger from AA.