In early recovery, you are very fragile and your thinking is still fogged. I understand the wanting to be with him and being unable to let go completely. But when we are in early recovery, just as addicts in early recovery are advised, we do best to "follow direction" from long-time recovering people or from counselors, because, just like the addict, our "best thinking" got us to this place of misery. Like the addict, our thinking is fogged and we are often incapable in early recovery of understanding or controlling our compulsions and emotional drunkenness.
I agree that going on this trip places you in an emotionally volatile situation which could possibly traumatize you even further.
But if you decide to go and isolate yourself with this active drug user who has damaged you in the past, take an Al-Anon phone list with you and use it, check in with SR daily, and do not have unprotected sex.
Above all, do not feel shame for any choice you make which you later come to regret. We are all learning, and we have all made wrong turns of one kind or another. If you decide to go on this trip, and if it is a catastrophe for you, do not feel shame. It will simply have been the outcome of being a wounded person who made a wrong turn.
And we all know what that's like.