This sounds like unfortunate circumstance to me, more than any effort to manipulate or dupe you on his part, and taking his wishy-washiness personally feels like a disproportionate response. But I think it says a lot about how ready for the give and take of a relationship you might or might not be at this point.
This is disappointing, and something of an ego bruise, but ask yourself, if he does decide to move away, will you regret not having spent a little more time enjoying his friendship? Or does your instinct say he only wants to hang out more to see if he can push that boundary a little further? It's okay not to know, this stuff is complicated, and it's totally okay to step back while you work it out. If he does decide to move your kids will come through it, with guidance from you not to take it personally.