Funny no one has talked about this before
I guess I always assumed it was a non-factor for everyone but me, but I sure am camera shy (on either end of the camera, actually).
I feel a little funny pulling over, but I'll do it if I'm taking a shot of something public (not a person or someone's house), and only if I can do so safely.
In Taiwan, I overcame my shyness a bit and took a few candid shots of locals, but in Taiwan everyone knows on sight that I'm a goofy tourist anyway.
Other than that, I tend to be quite intimidated taking pictures in public, especially of people.