I would agree, you should try and free yourself from this situation.
I know that where I live in UK, if I had any concerns for an adult, Social Services can be called. This could be an option for you, especially if he is being exploited.
Sorry you find yourself faced with all this.
Much Love
Bute x