Maybe the reason it hurts so much is because he IS a flaky guy. You don't feel your daughter is safe with him. That pretty much says it all. If he is hurt by seeing it on paper, that is his problem, not yours. As long as you were truthful, you have nothing to feel guilty about. His choices, his consequences.
I may sound harsh, but I think you have made the only sane choice in choosing your baby's safety and well-being. Way to go!
L