did you ask him to leave previously, or was it that he left to be with ms. so & so? i think it was the latter (reading between the lines in your post).
i would think (right now this moment, not anything from the past), that now he will need to be shown that
1. his behavior is unacceptable and he needs to leave the family (both the drinking and
the extracurricular relationships)
2. yeah, well, it's not so nice, but truly acceptable to behave this way toward you, and inside of a marriage and family.
either one or the other statement will be true, depending on how you react. the ball's totally in your court.