Assuming you're not taking a long set of fast burst images, the shutter is likely annoying only when you're somewhere super quiet, like a church, or library, or at the back of a hall where the speaker has no mic. I know I find the focus-confirm beep being much more annoying than the shutter. Perhaps a noisy focus can be an issue as well?