There's not "an issue" here--this is Dysfunction Junction.
I'm assuming there is something about him you find attractive, but whatever it is, it isn't worth the pain you (and potentially your kids) will experience if you stay with this guy.
I'd cut my losses sooner rather than later.