it sounds as though you have made a decision in your heart, but are talking yourself out of it. it doesn't sound like you have a healthy communication style. sometimes when i write it helps to clear things up for me. maybe a list of what is holding you back? is it what others may think, or hurting his feelings, your children?
i know what it's like to be stuck. it's often worse to live in the indecision than to make a scary or painful one.