A pact with a marriage counselor does not trump self-protection. I doubt the marriage counselor is privy to the strange behavior you describe. Sometimes you have to look out for yourself even at the risk of letting someone else down. And I would not be one bit surprised if your AH is counting on you keeping that "pact" and using it to further his own self-interest.
L