(((HUGS)))
WTBH, in my experience (personally and professionally in court), when an attorney tells his/her client 'not to make it worse', the attorney usually advises not to engage in any communication with the other party unless in writing or through the attorneys.
Is it possible to do this?
Certainly you do have the women's shelter option. You can choose one in your county or state that isn't influenced by your town, if you feel 'known' by too many people. I do think, just from your comment of "short of killing you" that he's got enough serious mental issues for you to protect yourselves further. Even if you don't check yourselves into a shelter for actual shelter, there may be one in your county who offers good, affordable services and one with whom your attorney will work.
I know attorney's fees are a huge worry; however, it is possible he would be ordered to cover that cost anyway, in the end. If you feel you're not being represented properly, you do have choices.
Hang in there, as safely as possible.