A Law Guardian would need to be appointed for your infant. If breastfeeding, that would somewhat preclude the child from being removed from your direct supervision for prolonged periods of time. Generaly, a few hours at a time. No court would willfully try to remove or restrict a father from his own child.
If it can be proven he has a drinking/drug problem, you need to somehow get documentation of this fact. The court may require him to submit to testing (ala Brittany Spears!). At least that way, you can be assured he isnt under any influences when nurturing his baby.