My thought has always been, if I were in my son's shoes and I KNEW I wasn't doing anything wrong, ie drugs... I would offer to take a drug test to prove that I was being truthful. When my AS got defensive, it was just as good as an admission to me.
On the other hand, 23 year old men do stay out late for other reasons than doing drugs... I think it's up to you what you want as your boundary, if the late nights don't sit right with you, it's well within your right as a parent that is providing shelter and food to lay down that rule. Like Ann said, it doesn't even have to be about drugs, simply the fact that we don't need to be waken at all hours of the night is enough of a reason to insist on being home at a set time. It's all about sticking up for yourself and the right that you have earned to live in peace...