I would have to agree the worst is behind you.
Allow yourself time, it will get easier.
It is so difficult when you have a child. I would limit communication to emails/texts. Say what you need and be done.
The kids have one healthy parent and that is you. The child is only 4, I am certain you are able to meet all her needs. Stay in your today. As long as I held on to a parental expectation of my ex-husband, I kept stumbling and hitting a brick wall. I had to accept it was my expectation and not my child's expectation that was causing me so much pain.
We were no longer a two parent household, and if he was missing out on things, it was because of his choices, and actions.
Sending you strength and wishing you peace.