I don't think you're a horrible person. However, I think your emotional affair is causing as many problems in your marriage as his issues.
I think finding comfort with someone else when you are in a situation that is not good is just a means of avoiding the work you have to do on yourself.