There are a lot of things that are harmless and legal but that are still disrespectful. If you played with a rubics cube or sat chomping on popcorn both would be legal and harmless but show a lack of respect for the people speaking.
I agree. I would be distracted, at the very least. I imagine you might get the same reaction in a church, in a doctor's waiting room, in an airplane or at the gym.