I don't think there is anything wrong with rushing to say no. They are your kids and you are right to be cautious. I too don't let anyone except those I know well take care of my kids. Kids can't protect themselves so that's our job. I am all for giving people a chance to redeem themselves, but my kids aren't going to be the ones who I test that out on. I explained this to my in laws a few yrs ago when I explained why it was I was not going to leave my D's alone with them. FIL is an alcoholic who flips flops from sobriety to not and MIL is the dictionary definition of enabler. They didn't like that. My H didn't like it either since it required the status quo of "let's pretend the 'he's a good man and doesn't mean to harm anyone' mentality" was being challenged. It did not earn me brownie points in the family. And I could care less. My kids matter more than the feelings of adults who've made choices and choose to continue to live in a way not compatible with what I think is safe for my kids.
You do not ever need to apologize for putting the kids first.
Your H has no reasonable right to expect you to trust him or a stranger he knows at AA (or a stranger he knows from anywhere) anymore than you have a right to expect your H to be trustworthy and reliable.
It might sting a little to him but your hesitation doesn't come from out of left field and his first concern should be like yours- your kids. Not the feelings of someone in his AA group.
Don't second guess yourself. Your mom instincts tell you what you need to know. I will never question what I feel when it comes to my kids and I don't think being overly cautious ever hurt anyone.