IMHO:
He should be happy, but he keeps calling you and telling you otherwise. I bet you're better off without him. For a moment, put yourself in her shoes and imagine him calling his ex saying those things about you and your baby.
It's Spring! Put on something pretty and get out into the world where you just might meet someone better. Paint your toenails girl and stick them out in public! You're wasting these beautiful days and loads of energy wasting your thoughts on him.
Just my opinion.