People on SR often give the advice to "play the tape all the way through" and that has always helped me.
I left my exafb when our daughter was 6 weeks old. I returned to him when she was 8 months, I left again when she was 22 months. I wish I had played the tape through and never gone back. I needed to help myself at that time and instead I continued to focus on him.
It is a really hard decision. Stay strong.