Have you investigated whether the LAW could set those boundaries? If you have exclusive possession of the house, you are completely within your rights to say he isn't welcome there without an express invitation. Look, the guy won't even honor your wishes to be gone when you call him and ask him to leave before you get home.
There is no "provocation" in setting specific, appropriate times for him to spend time with the kids--provided the kids are willing. Seems to me you are not honoring THEIR wishes, either.