Great job noticing for yourself before you get in too deep. All the flags are there Buddy, pretty soon you'll be trying to hide how much you really had to your wife. That's when the real fun begins... and ends.
Best to try and stop now while you still can.