I echo Splendra and Minnie, you need to get away from him and you also don't owe him any explanation as to why.
Just.............RUN!RUN!
The situation is totally in the negative for you. Don't worry about them or their marriage.
He made his bed he can lie in it.
Ngaire