Boy you and Mr Moose are really between a 'rock and a hard place' on this one.
Is there some way you could broach the subject gently with the GF that grandson really does not want to leave you when you take him back to your mom's house. It is obvious with no proof that calling CPS wouldn't help and might hurt the GF who is apparently trying real hard to better herself for her and her son.
No other ideas at this time, but am sending good thoughts and prayers your way, and have added you, Mr Moose, GF and grandson to the prayer circles I belong to.
Love and hugs,