How far are you moving? If its within the same state, definately do not give it another thought.
Move.
Has he filed anything yet, and a hearing set? He'd have to move quick to get a temporary order to keep you around before the hearing. And usually judges love mothers bettering them selves and returning to school.
If its a different state, or just to be sure, call a few attorneys see if you can get a free or low cost consultation as to your rights as the primary care giver. That way you know exactly what you can expect and his threats will be nothing more than words, because you'll know the real deal.