I think *everyone* understands.
They just don't approve, for a number of reasons. Here's the biggest one, probably:
Those that care about you don't want to see you get hurt. It's that simple.
If this guy loved you, he wouldn't be "fooling around" with you. He'd have left his wife. Period. But, then again, he said he loved his wife, didn't he? And look what he's doing to her? Something you really need to think about...cuz that's what your friends are concerned about.
You don't want to let go cuz what you are getting is more than the cost you are paying -- thus far. The bill hasn't come due, yet, white rabbit.
But, it will...
Shalom!