Thanks Anvil, Hanna.
I feel good about taking back the key for now. It feels in a way like it is a step back in our relationship, but in truth it is. He broke the trust, and it does have to be "earned" back. I do feel like I have become more aware of what lies ahead for him (and for me) if his ways dont change.
I feel like my house gave me a high -five. It will be much less likely he would even want to keep drugs there (even if he relapses) because he wont have easy,open access. If he were to be pushy to come over and make a bee line for under the sofa cushion, or one of my shoe boxes - I hope I would get suspicious!