What he's doing that's "wrong" affects you only to the extent that you allow it to. It doesn't have to have any more to do with you than what some guy in Texas ate for breakfast this morning. You keep putting YOURSELF back into his orbit. It's like walking into the street and saying it's unfair if a car hits you.
Just stop it. It really is that simple:
Stop It.