Why don't you just leave? If the outlook on your relationship looks like this, just do yourself and him a favor and call it quits. It's not fair to either one of you and if you hold on to these feelings, it sets the mood for a doomed relationship IMO.
Just leave. Cut your losses. Go. I'm sure he will survive without you. Don't wait for him to relapse...