If I imagine myself in your shoes--
I would feel too stressed with daily contact. I would want some breathing room. I would want a day or two or three a week where I knew that there was going to be no contact, and that I could completely relax and breathe easily on those days.
I think daily contact is hurting your healing.
Perhaps a discussion that daily is too often, and that you would like a couple days a week where it is just you and the kids. He probably won't like this, as he is trying to get back in the house with you.
But you need some breathing room, and you're not getting any.
Take care of you this way.