He has no legal right to live there. I would ask for the key, and, if he does not give it back, I would change the locks. He is 38, old enough to find his own shelter, his own way, but if it makes you feel better drop him off at the shelter.
I hope that you can find the strength to do the right thing for you and him. Let him fly and take care of himself.