Have you apologized? This journey is difficult and it's understandable we stumble along the way and make mistakes with our words and actions. Please go easy on yourself. Chalk the experience up to a mistake and then move forward.
I guess what would be interesting is his response to your apology. Will he accept your apology, forgive and let it go OR accept it but use it against you (constantly remind you of what you said)? I only raise it since the latter is not out of the realm of possibilities in early recovery and his departure did come across as abrupt.
I hope you both can move passed this incident and continue to work on your respective recoveries and marriage. Hang in there.