I understand.
Two things, you could probably go to a walk in clinic to get something to take the day of your appointment (maybe even the day before as well) to calm your anxiety.
If not, force yourself to go to the appointment anyway, at least you will have made a contact. You can explain to him that you have great anxiety, not just a bit but a lot and you almost didn't make to the appointment because of it.
Don't want to miss the opportunity to talk to someone that has connections! If all you do is go there and talk to him, that is a huge accomplishment.
I mean even if you aren't willing for him to address your problems, you can still talk. Always remember you are able to get up and walk out at any point, it's not a prison!