I don't know. I would think the same way you are. I would trust him, believe that the relationship is only what he told me it was. Let him know how I feel about it, of course.
But another part of me would want to tell him no more contact. But then I'd wonder if it's because I don't trust him or if it is my own insecurity that is causing me to expect such a thing.
You say that he goes out of town to spend time with her and these other folks from rehab. Is it your own insecurity that causes you to feel the way you do or do you really just not trust him yet (which IMO you have every right not to at this point)? Think about it and decide how it is you feel. Tell him you feel this way and ask him if he'd be willing to invite them over to your home for a day. They can sit and talk, do whatever it is they do there here at home. Maybe once you see how he interacts with her you'll have a better feeling about it and you can take it from there.