I would definitely see another attorney about getting visitation changed and protecting your children.
I speak as a grandmother who had to give up visitation of my grandson because my son could not be consistent and failed to show up when he was supposed to and caused emotional pain to the child. My son lived with me so that when he had to give up visitation, that included me.
That hurt...but in my heart I knew it was what was right for the child, and that was the most important thing of all. I cheer my son's ex for making such a harsh decision. In her case, the courts were not involved, they didn't have to be. She set the rule and we obeyed.
Please do what is right for your children, keep them safe. Everyone else is an adult and can deal with the repercussions.
Hugs, from my heart to yours.