I'm no mind reader but here's my interpretation:
He called me and says he still wants to help with the baby and asked if I would let him visit her for an hour or so a couple of times a week.
You have a need. He KNOWS what that need is and he is exploiting that need. "Help with the baby" is your hook. It is how he hooks you. Don't fall for it.
I feel guilty because I know he is attached to her
You cannot KNOW whether or not he is in fact "attached" to her. Feeling guilty about this is only going to hook you back in. Stop feeling guilty about ANYTHING. It is not your fault he is doing what he is doing---there is NOTHING to feel guilty about.
My vote is: DO NOT FALL FOR THIS.