When my husband was in active addiction he had an affair, for a long time that felt like the worst thing he did. We worked through it in marriage counseling ... and just giving it time.... He went into recovery also, almost 18 months now. NOW I feel like the worst thing that happened to me/us was the fact that he was in active addiction while I was pregnant, he was living on his own, I purposely left him out of the pregnancy, he missed the birth of our first child. The marriage we were able to restore, but that experience and being able to share it together as first time parents, we will never get it back.
The comments by your husband, I cant judge of course, but I know based on my experience it takes someone in recovery a long time to get over the shame, regret, and bad feelings about themselves that built up over their time in active addiction.
It is a lot for you to take on, you have been very generous and forgiving towards him, but its something that only you can make a decision on, if its worthwhile.