No, its not crazy, its realistic. Taking a financial hit is something to consider very carefully. But instead of thinking its a BARRIER, try to reframe it as a CHALLENGE to over come. What could you do differently in your life to afford to take a hit? Can you downsize? Find care for your kid that is free? (i.e. child care swapping - did this a lot in the early years, even though it meant my weekends were spent watching other people's kiddos) Sell an expensive car for a cheap one? Cancel cable? Qualify for any support services, i.e. child care, etc.?
Look at it from that standpoint - what CAN you do to make it work, instead of it simply being CAN'T.
And lastly, as a single parent, you owe it to your kids to be happy yourself. A happy Mom is an engaged Mom.