I have found that there will always be people who annoy me. I have two choices. I can make a confrontation out of each annoyance and make everyone as unhappy as I am...or...I can tune out the people who annoy me and keep my serenity.
Your feelings of wanting to spend time with your brother are valid. And since everyone else wants to spend time with him too now before he leaves, my thoughts are to either just share him and it doesn't matter who sits next to him or tell him you'd like to go for a walk or for a coffee with him and have an hour of one-on-one sometime before he leaves.
Hugs