Good job! I also like how you handled his "condition": "Of course." Once you are safely out, you can do exactly as you please. Much better for you not to engage him in an argument or dispute whether you are coming back. Let him think whatever gets you out.
Your counselor is right on the money, too. You look at anxiety, pay attention to anything that requires you to adjust your actions, and then set it aside and move on.