I feel like I'm missing something here. He doesn't *have* to stay at your house does he? What's preventing you from changing the locks and telling him no?
I'm not trying to pressure you or be mean, and I understand you feel like he's pressuring you and threatening you, but is there a reason you can't say no?
What did the lawyer say?