My difficulties have been basically two things:
1. I loved and miss getting high. My doc is pot, but I have tripped many times, too, and had many good times on mushrooms and acid. But my desire to use again is tempered by where I ended up from smoking pot every day: isolated, depressed, and in a job and home that make me unhappy and disappointed. At the end, smoking just made me paranoid, alone, and a little insane. From your post, it doesn't sound like you're at this place, but make sure you are being honest to yourself about all of the good and bad that results from using.
2. Sober reality. This one is harder for me to deal with. My feelings about my current lot in life are no longer numbed, and I have to face the regrets and mistakes I made as a a result of being a career pothead. And whatever issues I had as that made me want to be high all the time way back when I started, these haven't gone away. I really understand what you mean about trying to replace that feeling-for me getting high was my reason for living and getting through the day, and the spirituality, and sense of being in the now (, although completely artificial and false,) that came with it, are gone. I'm working the NA program, but I'm not sure if that will do it for me or not.
Pot and psychadelics are tricky, too. They don't seem as blatantly harmful as "harder" drugs, and a lot of people use them without any problem, so that can cloud my judgement, too.