What a mess. I don't have any kids but I can tell you that this "child" is 23 so is an adult. While your wife would likely disagree, he is old enough to live on his own. If he cannot get a job he will end up on the street or back in prison.
I would stand firm on not letting him come home, or, if you have no say (who's name is on the deed anyway?) then moving out is a good idea I think. Yes, it means your marriage but, honestly, won't having him live with you cost you your marriage anyway?