That right there is a huge worry for you i'm sure and I feel for you on that.
You know, it's kind of inevitable (starting to pick up negative ideas). I'm sorry to say that but they are "half" his, half his genetic makeup half his influence (ok maybe a bit less than that since he only has visitation).
The great thing here is that you, as the custodial parent, have the greatest influence on them. You get to be the balance. You get to be the one that talks to them about the great risks of using drugs or alcohol, about how gangs are not cool or something to be admired about how hurting other people is wrong etc etc.
It's a huge responsibility and I'm sure not one you intended to have to do on your own, but that is where you are and I know you will rise to that challenge, it's clear how much you care about your children's well-being.
I also completely get trying to unravel some of the "mystery" about who you married and what the hell happened. He kind of fell off a cliff there, he was there then he wasn't. That's all very hurtful and the prelude to all that was hurtful and I know that the mind likes to review, so I get that.