I still don't feel it is our right to set up certain limits to him but I am starting to see it is our right to also set up certain limits.
You're contradicting yourself here. Not only is it your RIGHT to set limits and boundaries for your own life and your own home, it is your RESPONSIBILITY to do so. It sounds to me like you are willing to set limits for HIS sake, but not your own. Nothing you describe about his behavior sounds healthy to live with. Please examine how he is affecting you, your life, your feelings, and your stress level. No, you should not be afraid of the person you are letting live in your house. You need to honor yourself here
Pogostick.