Have you checked with your family court to see if they have a location for supervised visitation? Some places do. There are also services that will provide a supervisor for a fee.
I think you may need to revisit your parenting time order. Maybe put in a provision that he is not to come to the house for visitation WITHOUT a supervisor. You shouldn't be put in the position of having to spend time in his company.