My advise is not depend on someone who you suspect has been drinking that day to take your child. That sounds like a very toxic thing for your child.
I would let him know you need to work out a parenting plan that says she can go with him on certain times providing he has not been drinking. Meet at the police station to exchange child. If you suspect he has been drinking he has to test. He cannot take said child until he has stable housing with utilities on and in clean living shape for child to be in.
If he splits and is not willing to do these things, you have your answer.
I am sorry. You definitely do not need to have him hanging around your home, that removes the stability of your own home for you and your child.