My thoughts on your situation
are this. You are a mother. Your first priority is to protect your child. You didn't say how old she is, but it doesn't really matter. Being around her dad and his girlfriend drunk and doing drugs isn't healthy for a child of any age.
Who cares what he thinks? Obviously he doesn't have good thought processes or he wouldn't be doing this while she's visiting.
((((HUGS))). I'm sure someone who's had similar experiences can offer better help, but as a mom myself, I wanted to state that your child's safety should come before your ex's feelings.