I had these same feelings earlier this week.
I spoke to my boyfriend who has been living in a sober house and he seemed a little off, a little depressed. I was worried that he was thinking about using, which I figured would eventually lead to him actually picking up heroin and using again.
I got myself so worked up about it and so worried and then I realized...it's out of my control!! There's not a damn thing I can do about it! It's HIS recovery, HIS decisions to make, HIS life to control...not yours.
Stay strong.