I can't remember what state you're in but in California it's very difficult to get parental rights revoked. It basically needs to be that damage has already occurred and it needs to be blatantly obvious. The state could care less whether your child is in clear and present danger emotionally.
My suggestion is to petition to change who supervises the visits, your XAH will likely have to foot the bill. But with a recent rehab (potentially) under his belt, you should have grounds to state that her grandparents clearly don't have a handle on things as well as they would have it seem.