OK, he's not moving far away--so you can visit, OK?
I am happy you were able to take the step to ask him to find a place to live. I know it was not easy for you. Good job.
Now, maybe take the step you need to take care of YOU. That can be hard but once you do it will be much easier to set your boundaries and start to get rid of those feelings of guilt. You have nothing to feel guilty about.