Accepting that he will continue to use isn't enabling - it's your admittance that you can't control his behaviour.
However, him coming to your house at 1am? You have every right to tell him to tell him to leave.
Cut yourself some slack. Figuring out what's 'detachment' vs 'enabling' is something I'm still trying to figure out myself lol.