I think you are wise to put what is good for your child first and it's great you have your family support to back you up.
He may be on methadone and trying to get/stay clean. .but he called you stupid and is not respecting your right to decide what is best for your child and you...that's selfish and a big red flag, from where I sit.
The more you work on yourself and set up a good situation for your child, I think it will become clear that this man is not good father material but just someone who wants to be in control.
I hope you find some peace with this soon.
Hugs