I agree asking all of them (or just her) over for coffee, and see his reaction, would give you a good pointer on what is really going on. Although for me "a kiss that just happened" is a deal breaker. No brownie points for "honesty", right now I am just tired of mind games. I have not made myself respected for a while, sorry I am no longer in for more heartbreak. You are either there 100% or you are no longer welcome.
What if you go on vacation and accidentally kiss someone? And you say "oh we shared some bad times, it just happened" then you talk a lot and spend time with the man away from your husband. Would he be OK with that????
Only my 2 cents...