If you want to allow him to come over, that is one thing. Showing up at 2:00 AM is quite another. Perhaps letting him know that after a certain time (say, 10 or 11:00) he will not be allowed to come in might help, but then you would have to follow through and not answer the door if he comes by later than that. Sometimes, boundaries need to be made into rules, but you will have to enforce them or they are useless.