You ARE doing the best you can for your kids!
When I look back on my childhood- I do not look back on how many birthday parties, or lego lands, or disneylands I went to. I look back at the quality of the time (which in my case was virtually none).
Kids don't remember money, or trips, or playdates. They remember if there parents were kind and loving and if they were THERE for them.
I mean YES, of course they remember trips and such... but that isn't the defining factor of their childhood.