FY, you've taken the first step of wanting to leave the relationship. That's important, even if you don't currently have the tools to take action.
I'm guessing he fills an important part of your life that you're missing in your marriage. Maybe concentrate on that thought, and also try to look at him objectively as a human, not something on the pedestal you gave him when you fell in love with him. Use a counsellor if you can, to clarify your thoughts.