Whether he doing drugs or not, if, he is not capable of watching a 2 year old, I
believe you need to make other arrangements for her care and safety.
You would not be taking her away from him, you would be taking her away from a potentially dangerous situation. She cannot take care of herself at 2 and should never be left alone.