My mother has a friend that is a family lawyer, so I talked with her. Since it is so close to my move, and there is no court date or anything set, she said go ahead and go. The worst case scenario is that if he files here before I am in the other state 6 months I will have to come back for a court date. I have witnesses to attest to his ongoing drug problem, his prior agreement to the move, and that I have always gone out of my way to provide visitations.
I believe he is having a hard time because I recently, for my own sanity, set stricter boundaries. He used to come over to my house and spend nearly all day Sat. and Sun., but now we meet at a public space for much less time. He would not want to believe it, but the kids do much better with the shorter visits.