I know it's a scary thought, because you want to do the right thing in the eyes of the law, even if it's not really the best thing.....when you look at the situation.
If you were to stop the visits and refuse contact with him in any way....he would have to go thru the courts and file contempt of court charges against you.
This is the way my grandson's dad did with my daughter. (my daughter is the addict).
She kept missing visits etc. etc. etc.....harassing the dad. etc. etc. and she didn't file with the court....she waited for an upcoming hearing where she said he wasn't holding up his end of the bargain. Dad very calmly told the judge that her behavior was very disturbing and disruptive to my grandson and asked if a guardian ad litem could be appointed? He additionally asked for court ordered drug screening of my daughter and anything else the guardian ad litem thought might be appropriate.
In other words....he ASKED calmly for assistance. What happened was....my daughter showed up for drug screens....which came back dirty.....and the guardian ad litem dealt with the court....not my grandson's daddy.
It took all of the drama out of all of it. And the state paid for all of it.