I am glad you are putting yourself and your daughter first here.
At some point, if you haven't already, you might want to get a court order to restrict visitation. That way if he breaks it and tries to see her when you say no, he can be arrested.
I'm sorry you are stuck in this sad situation and glad your daughter is too young to really understand what is going on.
Hugs