5dIII review at DxOMark

[HDNitehawk]

I don't get the D800's high score for Sports (Low Light ISO) performance. I didn't see where they offered graphs or anything to back this up. I would have thought the 5D III would have been the victor here.

[Daniel Browning]

I don't get the D800's high score for Sports (Low Light ISO) performance.

Makes sense to me. The D800 has more read noise, but it detects 14% more light. So depending on exactly what light level (and tonal level) you measure, results will vary. At the 30dB chosen by DxO (which reflects the "common" photographer pretty well IMHO), the 14% light is more important than the read noise, so it's 1/3 stop ahead. At higher ISO (say, 25,600), the lower read noise of the 5D2 will compensate for having less light, and so it would turn out a better result.

I didn't see where they offered graphs or anything to back this up.

It's in the same place as all their other cameras, which is to say their slow, enormous, and labyrinthine web site. Personally, I prefer http://sensorgen.info/, which is simply a conversion of DxO data into industry standard sensor metrics such as read noise, QE, and FW. (They do it with a "view source" on the right URL, which gives you DxO's data.)

[neuroanatomist]

I don't get the D800's high score for Sports (Low Light ISO) performance. I didn't see where they offered graphs or anything to back this up. I would have thought the 5D III would have been the victor here.

I'm going to repost something I put up on CR (modified slightly):

Honestly, everyone should take a step back and consider what the DxOMark scores mean - and don't mean. They are quite transparent about their scoring metrics and methods, and there's no reason to suspect there's collusion or favoritism occurring. At the same time, it's important to remember that the sensor score is just that - a score of the sensor itself, not a 'camera score'.

Furthermore, keep in mind that all of their overall scores are based on reducing the image to 8 MP - thus, the greater the starting resolution the more downsampling, which means lower apparent noise. That lower apparent noise means apparently better ISO performance and apparently more dynamic range (lower apparent noise lowers the 'floor' for the DR estimate). If you like, you can substitute 'artifically inflated' for 'apparently'.

They have a page describing the mathematics of the normalization to 8 MP, and on that page, there's the following statement:

What should be remembered is that doubling the resolution adds:
3dB to the normalized SNR
0.5 bit to the normalized DR
0.5 bit to the normalized TR
1.5 bit to the normalized CS.

So, compared to 8 MP the D800 is 4.5-fold higher (just over two doublings), whereas the 5DIII is 2.75-fold higher (just over one doubling). What that means, mathematically, is that the D800 has approximately one extra doubling of resolution relative to DxOMark's 8 MP normalized value - that accounts for all of the differences in the scores for both ISO and Color Depth, and part of the difference in dynamic range.

Note that DxOMark does provide the non-normalized data, they just don't use those data to calculate the overall scores, the rationale being that normalizing to 8 MP allows appropriate comparisons. In one sense, it does - if you're going to print 8x10" images all the time, then their scores actually apply pretty well.

Let me give a specific example for color sensitivity, which is the basis for DxOMark's Portrait Score. That's one area where the D800 with 25.3 bits 'beats' the 5DIII with 24-bits. In the comparison without normalizing to 8 MP, the D800's advantage pretty much entirely disappears.

Rick, that piece about reducing to 8 MP is what makes the D800 'better' on their Sports (Low-light ISO) score - the higher MP of the D800 means relatively greater effective noise reduction when reducing from 36 MP to 8 MP, compared to going from 22 MP to 8 MP.

This is why I quit reading DxO: http://www.dxomark.com/index.php/Cam...(brand2)/Nikon

I don't see how these measurements give one sensor a score of a 95 and the other a score of an 81.

Here's another example: http://www.dxomark.com/index.php/Cam...brand3)/Pentax

Notice that the Pentax K5 has scored an 82 overall; however, it is inferior to the 5d Mark III in ever category except Dynamic Range at low Iso settings. Thus, am I to infer that Dynamic Range at low Iso settings is the only measurement that matters when rating a sensor?

Again, it's about understanding what the measurements mean and how they're derived. It looks like the Landsacpe (Dynamic Range) and Portrait (Color Depth) have a much greater impact on the overall score than Sports (Low-Light ISO), and that actually is the case - for a reason. The overall score is an 'average' of the three use-case scores, but two of those three, Portrait (units are bits) and Landscape (units are Ev) are on log scales. ISO is a linear scale, so for example, comparing the ISO (Sports) values for the K5 and the 5DIII, the 5DIII 'score' is almost double that of the K5 (2293 vs. 1162), but when you log transform that difference, the difference is less than one stop (Ev).

________________________

To summarize, one key point about DxOMark's scores is that they are all normalized to an 8 MP image, and this strongly impacts the overall and use-case scores, giving an advantage to sensors with higher MP counts. Likewise, since DR and Color Depth are measured on a log scale, they have a relatively greater weight in the overall score than low-light ISO.

Try the following: click on the first link above (5DIII vs. D800), then in the comparison click the Measurements tab. I'd recommend skipping the ISO Sensitivity plot (it has nothing directly to do with the Sports/Low Light ISO score, despite the name of the test - it's really looking at ISO accuracy relative to the actual International Organization for Standardization criteria). But...look at SNR (signal-to-noise ratio), Dynamic Range (basis for the Landscape score), Tonal Range, and Color Sensitivity (basis for the Portrait score), and for all of them, look at the Screen plots - those are the data that are not normalized to 8 MP (vs. the Print plots, which are normalized and used to generate the overall and use case scores). When you do that, you'll see the following:

• SNR - 5DIII wins (when normalized they tie)
• Dynamic Range - 5DIII loses up to ISO 1000 but wins at higher ISOs (when normalized, the 5DIII loses up to ISO 1600, then they near-tie)
• Tonal Range - 5DIII wins (when normalized they tie)
• Color Sensitivity - they tie (when normalized, D800 wins)

So, for all of the above measures, the higher MP count of the D800 gives it an advantage when downsampling the images to 8 MP.

Bottom line is that I think DxOMark's measurements are more useful than their scores, but even their scores are useful - as long as you understand how those scores are generated, and the inherent limitations and caveats about them. The Overall Score is something the have to have (how can you have a ranking site and not actually actually assign ranks). It's like looking the Gross Domestic Product by country, and concluding that the USA is the best country in the world simply because it has the highest GDP. Or, to use a photographically relevant analogy borrowed from Bob Atkins, it's represeting the Mona Lisa by it's average color value.

[Daniel Browning]

Honestly, everyone should take a step back and consider what the DxOMark scores mean - and don't mean. They are quite transparent about their scoring metrics and methods, and there's no reason to suspect there's collusion or favoritism occurring. At the same time, it's important to remember that the sensor score is just that - a score of the sensor itself, not a 'camera score'.

Agreed.

Furthermore, keep in mind that all of their overall scores are based on reducing the image to 8 MP -

OK, after this I pretty much disagree with the point you're making here. Please indulge another one of my car analogies to try to help make it clear why I think you're driving after the wrong point. What would you think if I said the following:

Keep in mind that all of Motor Trend's road noise scores are based on driving the car at 80 MPH - thus, the greater the top speed the more you can slow down from the top speed, which means lower apparent road noise. That lower "artificially improved" road noise means apparently better accoustic performance.


Note that Motor Trend does provide the non-normalized (noise at top speed) data, they just don't use those data to calculate the overall scores, the rationale being that normalizing to 80 MPH allows appropriate comparisons. In one sense, it does - if you're going to drive exactly 80 MPH all the time, then their scores actually apply pretty well.


At 80 MPH, the Cadillac with 40 dB of road noise 'beats' the Pinto with 60 dB. In the comparison without normalizing to 80 MPH, the Cadillac's advantage pretty much entirely disappears.

Wouldn't you think I was crazy? There's just so many things wrong with that. Yes, the Cadillac *does* have higher road noise, but only when you drive it at speeds the Pinto can't even dream of. At any *overlapping* speed, the Cadillac mops the floor with it. There's nothing "artificial" or "apparent" about it. So why count the *possible* higher speed of the Cadillac as a negative? It can do everything the Pinto can do, and better. Plus it can do some stuff the Pinto can't.


Comparing sensor noise at 100% maximum spatial frequency makes about as much sense as comparing road noise at 100% maximum car velocity. (That is, none at all.)

[...], the rationale being that normalizing to 8 MP allows appropriate comparisons. In one sense, it does - if you're going to print 8x10" images all the time, then their scores actually apply pretty well.

No. You can pick *any* overlapping pixel count and the performance differences would be the same: 22.3 MP, 8 MP, 1 MP, it doesn't matter. So you don't have to print 8x10" exactly to notice the same difference in performance -- the difference is always there.


Furthermore, there are many additional benefits of higher resolution that DxOMark does not factor into their comparison. First, if you actually give people a choice between "less noise + less resolution" or "more noise + more resolution" (where the noise power per spatial frequency is the same in both), they will pick the latter every time. Even people who *think* they hate noise. Lower resolution images just look like high-res ones with NR applied. They are never as good.

Try taking any noisy image. Downsample it to to one quarter of the spatial resolution (thus reducing noise power significantly), then resample it back up to the original pixel count. Now compare it with the original. The downsampled one has much lower noise power, but I bet you *anyone* would pick the high-noise high-detail image as their preference.

On top of that, you also have to factor in the OLPF. At all spatial frequencies above 80% of the 5D2 (i.e. more than 14 MP), the contrast is going to be reduced by the OLPF, while the D800 will not (because it's OLPF does not kick in until much higher frequencies). To compensate you will have to either increase sharpening, which increases noise, while the D800 will not require any sharpening to reach the same contrast level.

So even if the D800 had the *same* score as the 5D3 -- that is, if you got the same exact noise level after downsampling to 22.3 MP, the D800 would *still* be superior in any comparison at 14 MP or above. Below 14 MP, they would be the same.