As I said, it's totally up to you how to handle things with him. But again, it seems you are equating one thing with another. Limiting his access to pushing your buttons does not have to equal banning him from the children. He can take them to a park, take them out to dinner, go for a walk with them, or whatever. Letting him spend time with his children doesn't mean you have to let him come over and walk all over you. And not wanting him in your face doesn't equal blaming him for your troubles.
L