You are in a bad spot. You do need to cut off ties especially for the baby. You could try blocking his number. I assume he shows up when he is out of money.He probably tries to convince you that everything is your fault. He wouldn't be on drugs if you did ......fill in the blank.
The beginning of letting go is the hardest part.