Mine always freaked out too. I think he was paranoid that I would discuss him and he wanted control over that.
That has been years ago and it never occurred to me until today that maybe he was flirting up some women or had other gfs there. In his case, that might well have been true.
I get that you would like to respect his wishes. That is very good and decent of you. But, it is impacting you in a negative way now...and you need to be in your office. I would go ahead and go to the meetings you need that are available to you wherever they are. You know him best..as to whether you prefer to tell him or not. If it is just going to be an argument and/or have you feeling guilty, I would skip it and do what I needed. He may just have to compromise. They are open to the public...and he is all ready seen there.