You are giving your son the best possible childhood you can, just by putting him first! I know exactly how you're feeling, because I go there, too, sometimes. But I try to remember that there are lots of kids from two parent families who have far from perfect childhoods.
Keep doing your best, and remember, when we look back on our childhood, it isn't the big stuff or the stuff we thought was big (at the time) that stays with us. It's the love, consistancy, laughter AND boundaries that define childhood.