You can do whatever you want until you are mandated by the court to adhere to specific rules. Does he have any kind of court document that says you can't move? No? Well, then you can move wherever you like! Of course you MUST check with your own lawyer to be certain, but it sounds like he's just bullying you every which way he can think of. Know your rights.