Interesting topic. I think it really depends on the location and situation, and how people go about it. Firing flashes in a restaurant, setting up tripods, and climbing on chairs is a nuisance IMO. Taking pictures of each other at the table without a flash, even with a bigger camera, shouldn't be a problem. I have done so myself, and on one occasion, the waitress asked to take a picture of all of us. She had to walk around some tables and climb on a chair because I only had the 50 on a crop camera - that was definitely more than I would have asked for, or done myself.
All in all I would say, the camera itself (even if it is big) should not be a problem. Shutter noise can be, depending on the situation and noise level. Flash and tripods and climbing on chairs or walking backwards into the neighboring table in a restaurant can be extremely annoying - I would refrain from using those in most situations. It seems that too many people just lack some common sense in this regard.