The unfortunate fact is that when you've got young kids, you can't just follow your dreams--at least not as easily as when you have to be concerned only about yourself. They won't be little forever. And you won't be stuck in this job forever, either.
You never know, this job may lead to other opportunities down the line that aren't apparent right now. Even if it's simply doing a great job for someone who will give you a glowing reference when you're ready to move on.
When do you start?