I think if it were me I would just end it, and not say why. He doesn’t sound like an equal partner or healthy complement, you don’t trust him anyway because you know of him as a person who won’t communicate. It sounds toxic and like nipping it off as drama free as possible is best. He gets to have his story, and you get yours. No hard feelings. Everyone moves forward with the direction of their lives that they have chosen.