Jabe, I feel for you. That's a tough spot to be in.
I think what you're feeling is completely normal. Setting boundaries is extremely difficult, especially when you haven't set them before.
I also don't know if I agree with others that light contact, in a supervised setting (or public) is a good thing.
The mother of my newborn (25 days old today) is an addict, and immediately after birth, she went missing and we can only assume she's been homeless. She somehow made it to visitation hours, but had lost an incredible amount of weight, had a black eye, numerous bruises/cuts/scabs, and track marks. I'd rather not see her like this and have that be the last memory of her if she chooses to go down this path.
I want to remember her as the beautiful, smart, caring and sweet mother of my gorgeous son, sister to 3, and friend to many. That's the woman I've chose to welcome back into my life. Until then, my boundaries are pretty set and I will only cross paths during court ordered visitation.