I'm so sorry for what you are going through. I can only share my ES&H. My niece was 1 year old when her mama died in a car wreck. Baby daddy had a long history of drug abuse and what went with it.
My dad and stepmother (niece's grandma) raised niece. There were lawyers (baby daddy parents fought for custody), guardian ad litem and therapist for niece by the age of 3.
Long story short, my niece is more like a little sister. I'm an RA and recovering codependent and now that SHE is a mama, she understands where I came from.
Somewhere along the line her bio-dad tried for custody and was told "you have a snowball's chance in hello, drop it".
She gave him a second chance on promises. Now that he is back in jail, has let her down repeatedly, she is DONE.
He is threatening you to scare you. Don't fall for it. If he wants to be "daddy", he will have to have a steady job, pay child support, and do random drug screens.
My family has dealt with this for 20 years, you don't have to do it. I know it's hard to let him go, not see his child, but your child and you are first priority. Take care of YOU!! He will do what he will do, and shows no indication of wanting to do the right thing.
Hugs and prayers,
Amy