Sounds fine to me. Just low-key and not a big deal. Just tell him you've made your own plans and you're sure he will have fun with her on his own.
You can minimize the interactions by not relying on him for things like rides, and by not accommodating his own requests for favors. "Sorry, I can't," is fine. You don't have to explain ANYTHING. The exceptions, as I said, might be when it serves your own interest in ensuring your daughter is safe.