I've become a big believer in reframing my point of view when I feel like, "Why me?" I also live in a little apartment with my two children and have felt the way you do when I see other families looking so perfect and happy. But when I feel like, "Why me?" I now try to ask, "What's the opportunity in this?" For me, the opportunity is to heal from the things in my past that have made me a dirt magnet. I can take my time to get to know myself and become solid in my beliefs and principles, so that I can quickly say "buh-bye" to dirtbags and say "hey-ayyy" to the good ones.
Having a happy marriage and all that is not my path right now. I did have to grieve that, and maybe you are going through a little bit of grieving right now. But your path is yours and it doesn't have to look like everyone else's.
For what it's worth, word got out about my H's affair, and I had several seemingly happy people on FB contact me and say, "OMG! Me too!" There *are* happy families out there, but it's not like it's everyone but you. I actually get a lot of joy from seeing the happy families out there. It's possible. I have a few different happily married couples who are my clients, and they met in their late 30s (usually online) after kind of giving up on the idea of having a family, and now they have wonderful partnerships and beautiful children! It's all possible, but maybe your HP wants you to heal yourself first.