Anvilhead hit it right on! If he gets stopped with drugs/reported they can take your kids away. It can happen just that simple. I agree with everything that others have posted. You need to focus on yourself and your kids. He is going to tell you anything to get you to believe him. Read the sticky "What Addicts Do". That is why you are questioning yourself.
Read the stickies, go to alanon, and keep posting. This site has been very helpful to me. There are people here who has gone through it all. They know exactly what you are feeling. Your fears are our fears. You are not crazy. Detach yourself from him and really think about your boundaries and their consequences. He will step very close and even cross those lines. Be true to yourself and your children.
You are in my prayers. Send a pm if you need.