I insisted on what is best for the kids. I went to a therapist, was honest and open about the situation, asked their advice on what was best for the kids and went from there to the lawyer's office.
In my case, we are week on, week off. She has to - has to - continue with her therapy until the kids are 18. We covered birthdays, holidays, pick up and drop offs, who pays for what, we've agreements on some limited contact with her family members that are questionable. I included all known aspects of her history (mental health, substance abuse, suicide attempts, past parental rights issues with her first husband, etc.). I went hard and fast to enter her history into the record and used that as leverage to get what was best for the kids - namely, my ability to "pull the plug" on them having time with her, should she not remain stable.
In response to her "He is this way and that way, a bad father etc...." I put all three of my kids in therapy right from jump street. I didn't sit in on the sessions, but did inform the therapist that our family was splitting up. I even let my ex take them to the therapist on her weeks with the kids so there therapist could get to know her as well.
The kids validated where the issues were in the family and they got some help with their issues..... best thing I could have done. At that point, I could have gone to court and proved what issues the kids have and with whom.
The main purpose of the custody agreement is to define the boundaries that are needed to keep me and my kids safe -- that is the position I took.
Best luck.....