This reminds me so much of the description of a narcissist and other people. To the narcissist, the rest of us are just mirrors. We reflect what he wants to see about himself, or he'll reject the mirror and find another flattering one.
Narcissist parents love young children because they usually adore their parents and want to please them. As soon as the child grows up and starts wanting other things, the narcissist will reject the child and/or make the child's life hell by berating them or making the standards for perfection so impossible it's crushing.
Being around a narcissist (A or otherwise) is so painful because being around them and being happy is completely dependent on being a cipher, a zero, someone who is unseen and unheard, except when it behooves the narcissist -- usually for an audience! You're bad for having any needs at all. He has needs though, and it's your job to tend to them. Suggesting he might need to change his point of view is enough to get you kicked to the curb, so you put up and shut up to survive. When someone else is around to impress, he might trot you out to put on a show. Once the audience is gone, he puts you back on the shelf.
I'd argue contrary to what your family was saying that maybe it wasn't you who he adored -- he adored what being with someone like you meant about him, and only as long as you would shut up and take whatever he was serving, and serve it back to him on a silver platter with a smile.