Has he ever done anything in their presence to endanger them? Is there a pattern? Just try and take an honest look at your motives. It is really difficult to sort things out sometimes. As long as he is "fine" when he is with them, what he does away should not matter, at least as far as visitation is concerned. We have to be careful that we are not punishing, because someone is not behaving the way we think they should or have wronged us.