I have zero experience here but I know I would have a hard time giving over full time custody under the circumstances that you describe. I'm glad you are in a position to just deny the request & move on.
The only time I've ever seen this dynamic IRL is with one of my BFF's sons.... She & his dad split when he was about 6 months & dad & his new wife went on to become champion beer-drinking alcoholics with very loose rules for her 2 kids & BFF's son when he would visit with them. At about 12yo, son decided living with dad would be awesome because it was just. so. much. fun! ... until the reality of living with 2 other unsupervised step-siblings set in & his struggles in school got to be more than he could handle without a parent holding him to some responsibilities. He asked to move back with BFF & she fought for him but by then dad was used to drinking away the amount he wasn't spending in child support due to the new arrangement & dug in & fought back hard. It went on & on back & forth & then he was 13, 14, 15..... and he never quite got back on the right path again & has continued to struggle now into his 20's. It's so sad to me, he was such a bright, talented child & he ended up SO angry & lashing out at everyone & everything around him.
I think you're right to want to keep him close, he loves you even when he's angry or has a hard time understanding & expressing it. It may be a few more years before he really "gets it" but for me I have no doubt he WILL eventually get there in his mind as he matures & sees the amazing role model you've been for him all this time. ((((hugs))))