Hi
Do not go easy on him.
Your daughter's safety is the most important issue here.
You need to know she is safe when she is not with you.
I would not be going for an agreement but for an order, then it can be enforced by the courts.
Bring all issues to the surface now & deal with them now.
Get legal advice from a good lawyer, sometimes the advice is free & you pay for any action the lawyer must take.
Best of luck.