IMHO.
He's putting the burden of responsibility on you. You didn't smoke crack so why are you shouldering the accounting burdens of his recovery, allowing out his money like hes a child. If you gave him too much money, or gave him his card back, he can then say "YOU gave me the money, i asked you not to!" and then its YOUR fault he relapsed (not really, but he'll say it). If he can say "well you didnt _____ so how can I _____", its not going to go well, and yes; its unhealthy.
Its his recovery. His responsibility. His decision.
Again, just my humble opinion.