Hey, I'm a farmer, it's a great idea! Seriously, it is. If, that is, it's something you would enjoy doing, it's extraordinarily rewarding work. Though work it definitely is. Whatever you do, if you do it with passion, you will find great reward in it.
Like you, fear has been a driving negative force in my life. Fortunately I'm not fearful of everything, mostly it's the reactions of other people that scares me to immobility and retreat. But over the years I've learned (still learning) that fear only has the power you give it. So what about what other people think. What do I think. If I trust my own judgment, and I do, I proceed boldly forth and act. It's only when I allow my fear to rule me that I fail.