Because of his friends, lack of license and talking to your child about things he has been asked to stay silent about...if I were you, I would arrange to meet him at a public place, like a restaurant, where you have people around to help you if things go south and where he can visit with his son under your supervision and rules.
If he fails to show up, I would not allow future visits even if it means getting a court order.
My own (addicted) son was not allowed to see his son after he disappointed the boy too many times by not showing up and it took an emotional toll on the child. I admire the child's mother for doing that and putting the best interest of the child first.
Good luck with whatever you choose. Just remember that you are the responsible parent here and whatever you decide is okay is how it will be...whether he likes it or not.
Hugs