it's good you can so clearly speak to what you "miss" and what you really were dealing with. it's tough when our idols fall off their shelves, and smash into clay bits at our feet. but it is critical that we DO see things are they truly ARE and also as they truly WERE.
Honestly, most of the good times were when he was high. He'd tell me I looked pretty, he'd play with the baby and even say she was "so cute"... think about that, he was HIGH when he was playing with your child. and you almost wish he would GET high again.........that is how far we will go in our heads trying to negotiate a way to live with the unacceptable.
you did the right thing. rarely is that the easier softer way.