Histogram / RAW conversion comparison

[neuroanatomist]

Hmm, that is odd. The only other thing I can think off is that there may be something wrong with the installation files. I know it sounds unlikely but I can not think off any other reason. Have never seen this before...I remember my wife had a problem with DPP a couple of years ago. The RAW tab or the RGB tab menus did not work properly. I had to do a clean install with "fresh" files from Canon to get it to work.

A clean install is a good idea, and might even work! So, if you decide to try it, let us know if that does the trick. [:P]


Actually, I'm serious (for once). Assuming we are correct in thinking the white area of the plot shows the theoretical dynamic range of the sensor, I suggest that you try it because I believe DPP on my computer is showing the correct data and DPP on your computer is not behaving correctly, for two reasons - one logical, and one empirical.


The logical reason goes like this: if we're correct that the 'white area' on the DPP histogram represents the theoretical maximum dynamic range of the camera, then an image from a camera with a 14-bit analog-to-digital converter (like the 40D)should have awider white area on the graph than an image from a camera with a 12-bit ADC (like the original 5D). (You can confirm the bit depths here.) So, the image below (not mine, as I said) would seem to be correct:








The empirical reason goes like this: I downloaded a Rebel XS/1000D RAW file from Imaging Resource, and the equivalent 7D RAW file (thanks for the link above!!). Opening both in DPP 3.8.2 on my computer, I see this (XS on the left, 7D on the right):


[img]/cfs-file.ashx/__key/CommunityServer.Components.UserFiles/00.00.00.35.15/1000D-RAW-histogram.jpg[/img] [img]/cfs-file.ashx/__key/CommunityServer.Components.UserFiles/00.00.00.35.15/7D-RAW-histogram.jpg[/img]


It seems that my DPP RAW histogram shows a 2-unit wider dynamic range for the 7D (14-bit) than the Rebel XS (12-bit), starting at -11 vs. -9. The 2-unit difference on a base-2 log scale corresponds to the 2-bit difference between the ADCs of the two cameras [since log₂(2^x) = x].


However, the screenshots you posted from 14-bit cameras seem to be showing the same dynamic range as the 12-bit Rebel XS in DPP on my computer - that's what makes me think yours is not displaying correctly. If you're willing, try downloading the same two files linked above, and see what your computer shows (maybe there will still be a 2-unit difference, offset to -9 vs. -7 in your case, and we'll be back to square one!).


I hope the above makes sense. Thanks for taking the time to stick with this, Johnny!


--John

[Johnny Rasmussen]

John,


You could be right of course. What I am seeing could be wrong for some reason.We know there is a huge technical difference between a 14-bit and a 12-bit file.Does it show ut in DPP...that is the question?


I downloaded both files from the link you gave mebut the DR is the same. I even checked my wifes computer and it is the same. She is using the previous DPP version, 3.8.0 for Windows.Then I downloaded the DPP manual for Mac, v3.8.2 and grabbed a screendump. As you can see itis different from what youare seeing, showing a DR from -9EV to around +3.6EV.


[img]/cfs-file.ashx/__key/CommunityServer.Components.UserFiles/00.00.00.22.85/DPP_2D00_3.8.2_2D00_manual_2D00_for_2D00_Mac_2D00_ page_2D00_54.jpg[/img]

[Johnny Rasmussen]

PS! Does the 7D file have the vertical bar to the left like the XS file did? If so can you drag it to the right and adjust DR?

[neuroanatomist]

As you can see itis different from what youare seeing, showing a DR from -9EV to around +3.6EV.

True. I see that in the manual. But then again, I see that when I open up a 12-bit image in DPP v3.8.2. We don't know what camera the image used for demo purposes came from (in the section on viewing the shooting information, whoever wrote the manual specifically redacted the camera model/image size/etc. - it's a crop body, though, since it's using an EF-S lens, and it goes to ISO 3200 so it's not an XTi, but it could be a 20D, which is 12-bit).

We know there is a huge technical difference between a 14-bit and a 12-bit file.

Certainly. Canon even published an article on the benefits of 14-bit A/D conversion.

Does it show up in DPP...that is the question?

It seems to...at least for me. RAW histograms from other 12-bit cameras (5D, XTi/400D) show up as -9 to +3.5 just like the Rebel XS/1000D, while RAW images from other 14-bit cameras (5DII, T1i/500D) show up with the wider dynamic range of -11 to +3.5 just like I see with the 7D.


I think if you don't see a difference between the two image types in the DPP RAW histogram, that's not correct.

PS! Does the 7D file have the vertical bar to the left like the XS file did? If so can you drag it to the right and adjust DR?

Yes, that moveable bar just initially overlaps with the left axis of the graph - hovering the mouse over that axis changes the cursor to a bi-directional arrow. The line can be dragged in from the left edge to adjust the DR.

[Johnny Rasmussen]

Interesting. I am beginning to think thatyou are seeing the true capability of the 14-bit sensor. Do you have some images from your own camera shot with HTP on? That should move both vertical bars 1EV to the right and give you -10EV (theoretical shadow DR) to +4.5EV.


Btw, it seems like Canon has the best cameras when it comes to highlight range. Turn on HTP and the 1Ds MIII will give you almost +5EV above middle gray. That is actually better than the (in)famous Fujifilm S5 Pro.


I still don´t understand the differences between the rendering of the histograms. Think I have to ask Chuck Westfall about that.

[neuroanatomist]

Interesting. I am beginning to think thatyou are seeing the true capability of the 14-bit sensor. Do you have some images from your own camera shot with HTP on? That should move both vertical bars 1EV to the right and give you -10EV (theoretical shadow DR) to +4.5EV.

I don't usually shoot with HTP, since the highlight preservation comes at the expense of increased shadow noise and the 7D doesn't really need any 'help' in that department. But, I just grabbed a couple of test shots -/+ HTP - the desk lamp against the closed curtain was intended to result in a histogram with peaks at both ends of the DR. The one on the left is without HTP, the one on the right is with HTP.


[img]/cfs-file.ashx/__key/CommunityServer.Components.UserFiles/00.00.00.35.15/7D-RAW-_2D00_HTP-histogram.jpg[/img] [img]/cfs-file.ashx/__key/CommunityServer.Components.UserFiles/00.00.00.35.15/7D-RAW-_2B00_HTP-histogram.jpg[/img]


As you can see, the DR 'window' does shift 1 EV to the right with HTP, compressing the shadow end and giving some additional headroom on the highlight end. It's worth noting that in the non-HTP image of this contrived scene, there are pixels in the RAW data that span the full dynamic range represented in the DPP histogram.


As an interesting side note, parts of this simple image can be easily resolved on the histograms - the peak at the far right is the bulb, and the next one just to the left is the lamp shade. The big mushy peak on the left is the curtains. It's interesting that there is a series of peaks in between (-3 to -1 EV), seen only on the +HTP histogram. They might represent the forward folds of the curtains that are brighter than the backward folds, and get progressively less intense further from the light - but it's certainly not an obvious visual difference between the images, so it's interesting that HTP resolves them. I wonder what that says about the algorithm. But then again, those peaks may be an artifact of the processing, which is probably intended for 'real world' scenes and is here being applied to something pretty artificial. Some noise reduction algorithms can actually create patterns in a blank region of an image. One of those peaks is at -1 EV, and there's nothing in the image that stands out to me as a patch of something one stop darker than middle gray - that sort of supports the artifact idea, but really, I have no idea what's going on there.


Regardless, in any case, HTP does appear to offset the DR to the right for me.

I still don´t understand the differences between the rendering of the histograms. Think I have to ask Chuck Westfall about that.

Great idea! If you ask him and he responds, please let me know what he has to say.


Thanks again!


--John

[Johnny Rasmussen]

John,


I received an answer. He said that Canon USA are aware that the shadow end appears to be longer on the Mac version of DPP but they have not received a technical explanation from Canon HQ. They say that the actual DR captured by a given sensor is the same even if the representation is different for Mac users. He also said that the white portion of the scale represents the approximate DR of the camera in use and that they do not have detailed information about the scale it´s using, it is referred to as highlights and shadows only. He further said "when attempting to adjust the left end of the histogram on the Mac version, the line for the deepest shadow seems to jump to the same starting position on the scale as the starting position for the Windows version of DPP."


What do you think? Any other ideas? Seems like it will not be easy to get to the bottom of this. I guess Canon HQ has it´s reasons not to publish all the details but I have no clue why.


Do you think this has to do different "country versions" ? When you install a new version you are asked to choose your location, (Europe, Asia, America ). Running out of ideas.


Since it seems like you have the camera and computer with the best dynamic range why don´t you ship it to me? [G] I volunteer to test it. I have a few portraits to make and better DR is always welcome..[]

[neuroanatomist]

Hi Johnny,


Thanks for checking that out with Chuck!

He said that Canon USA are aware that the shadow end appears to be longer on the Mac version of DPP but they have not received a technical explanation from Canon HQ.

I've got one problem with that explanation - it fails to explain that the same histogram as I see is also seen in Windows (at least, I think it's Windows...but it's certainly not the Mac version of DPP!):

He further said "when attempting to adjust the left end of the histogram on the Mac version, the line for the deepest shadow seems to jump to the same starting position on the scale as the starting position for the Windows version of DPP."

I do notice the 'jump' but I still wonder what's going on, since there are clearly pixels at both ends of my wider histogram. Since I see a difference in the width of the active part of the histogram between images from 12- and 14-bit cameras, and you don't, I'm still thinking that what I'm seeing is more accurate.

Seems like it will not be easy to get to the bottom of this.

Agreed.

[neuroanatomist]

I'm still thinking that what I'm seeing is more accurate.

I was thinking more on this as I drove into work this morning. I do think the representation I'm seeing (where the DR of a 14-bit image is displayed as being wider than the DR of a 12-bit image) is more accurate than what you're seeing, Johnny. But, I should point out that neither is truly accurate, and so the differences are probably irrelevant in a practical sense. As I stated above,log₂(2^x) = x, so if the scale being used is log₂of pixel luminance, a 14-bit image should have a DR of 14 units. The range of -11 to +3.8 (or so) is ~14.8, nearly a full unit wider than it should be for a 14-bit image; likewise, I see a 12-bit image displayed with a range of 12.8 units.


Regardless of the actual units, DPP is not 'throwing away' any of the dynamic range of the RAW image (until it's converted to an 8-bit format like JPG, that is). So, practically it doesn't really matter if the scale is-11 to +3.8,-9 to +3.8, 0-16,384, -20 to +20, or even if there are no units at all. So long as the pixels are binned across a range (and those bins are sufficiently narrow as to allow meaningful adjustments), the histogram serves its purpose.

[Johnny Rasmussen]

I came across a discussion in the dpreview forum. It seems like we are not the only ones that have been wondering about this. One guy said that the differences seen in the histograms is "canons way of showing us the wider DR of the 14-bit raw-files". He argued that a 14-bit file should be able to show a 2 unit wider DR. I went to Canon Japan, looked at DPP tutorials and they had the same wide DR as your version.


Anyway, you are right, even if the representation of DR differs it does not really matters. I can live with it []


I was thinking about something else, John. You mentioned you did not shoot with HTP on because of increased shadow noise. Is it really that visible? I have not shot with the 7D but it is hard to see any difference in files from the 50D. If I enlarge the image 200% I can see a small difference. Bob Atkins tested the HTP function http://www.bobatkins.com/photography/digital/canon_eos_40D_review_6.html#highlight and particularly looked at the shadows. The link is for the 40D but he has also tested the 50D and the 7D. He said that the differences between these models are insignificant in the shadow region. I know that sites like dpreview criticized the 50D compared to the 40D but I wonder if this is because of the methodology they are using.