Display Transform Based on JzAzBz LMS

Hello again,

some more tests on a Cornell Box. Here is the setup : linear-sRGB textures, rendered in ACEScg and displayed in sRGB. Here we go !

ACEScg render displayed in sRGB (ACES)

ACEScg render displayed in sRGB (OpenDRT v0.0.83b2)

ACEScg render displayed in sRGB (OpenDRT v0.0.90b2)

ACEScg render displayed in sRGB (JzDT)

And some luminance tests :

desaturated ACEScg render displayed in sRGB (ACES)

desaturated ACEScg render displayed in sRGB (OpenDRT v0.0.83b2)

desaturated ACEScg render displayed in sRGB (OpenDRT v0.0.90b2)

desaturated ACEScg render displayed in sRGB (JzDT)

I think the same observations can be made from the previous renders. I still believe there is somewhere some ground truth that we can rely on rather than subjective/aesthetics judgement. I believe this is what is hinted here and here.

Hope it helps… As always, nice work Jed !

Chris

1 Like

And finally… Some light sabers tests comparing the same display transforms.

ACEScg render using ACEScg primaries, displayed in sRGB (ACES).

Same render, displayed in sRGB (OpenDRT v0.0.83b2).

Same render, displayed in sRGB (OpenDRT v0.0.90b2).

Same render, displayed in sRGB (JzDT).

In the following examples, I completeley desaturate the scene-referred values.

Desaturated ACEScg render, displayed in sRGB (ACES).

Same desaturated render, displayed in sRGB (OpenDRT v0.0.83b2).

Same desaturated render, displayed in sRGB (OpenDRT v0.0.90b2).

Same desaturated render, displayed in sRGB (JzDT).

And for comparison, I have applied the desaturation after the Output Transform.

Display-referred desaturation in sRGB (ACES).

Display-referred desaturation in sRGB (OpenDRT v0.0.83b2).

Display-referred desaturation in sRGB (OpenDRT v0.0.90b2).

Display-referred desaturation in sRGB (JzDT).

That’s it for me. Happy rendering !

2 Likes

Hi @ChrisBrejon ,

I think it would be possible to get a kinda-ground truth for the LEGO sailor scene by reproducing it with physical legos and lights and shooting it with different brands of lo-tech point-and-shoot and/or cellphone cameras that all go straight to sRGB (with default auto settings). They all have their own different magic sauce recipes in order to try and get and image close to what you’re seeing but taking an average of those and trying to avoid getting married to one result or another could give us an idea w.r.t. how much light each colour of lego brick should be reflecting and what average hue it should have in Rec.709. Next, with the physical rig as a reference, one could take a raw photo with a high-end camera and do a grade from log to PQ/Rec.2020 @ 1000 nits on a reference monitor in order to try and match as close as possible what one sees. This could give us a ground truth reference for the HDR version.

Currently, @jedsmith 's DRTs are the closest that we have to a ground truth reference but we can’t know that as they were evaluated on reference images that are either full CG or pre-existing footage which none here knows what it looked like for real when it was shot. Essentially, Jed’s DRTs are almost perfect and we’re in the last 10% stretch where we need to tweak, adjust or just plain throw alternative algorithms that look similar but slightly improved at the wall until something sticks and everybody is happy.

I, personally, will be happy when emissive reds and blues (fires and skies) will be under control without sacrificing saturation in the diffuse reds, greens and yellows too much (skin tones and grass), i.e. when memory colours are under control. That doesn’t mean that we should bake in red->orange skew and/or blue->cyan skew and/or contrast to an excessive level though, just that those can be easily achieved with simple hue shifts that work well in all targets instead of requiring different LUTs per target. Jed’s perceptual correction is an example of a hue shift that “just works”.

1 Like

Can’t we just encode an image with display encoding and no highlights roll-off and think of it as this is our reference? For example to encode with power law gamma 1/2.4 and display it on gamma 2.4 display? All the colors that are not clipped are our reference. And the same for HDR. What am I missing? Or those clipped colors are actually what I’m missing?

Those clipped colours are indeed what you are missing :slight_smile:

Rec.709 at exposure level +2 or +3 will give values >1 very fast. Take a non-pure red with a bit of green in it and you will notice that it skews progressively towards orange as you raise exposure because the red channel has already been clipped to 1 while the green channel is still being raised. It will also get stuck on (1, 1, 0) which is pretty useless if your source has a higher dynamic range than 0…1 diffuse reflectance.

1 Like

Hi all!

I’m a random developer who fell down the fascinating rabbit hole of color management about 2 weeks ago!

Thanks to Christophe Brejon, Jed Smith, Troy Sobotka, Daniele Siragusano, and many others, I went from blissful ignorance to compulsive thinking about colors, day and night :wink:

I would like to share with you some thoughts I have about a similar algorithm to JzDT.

My understanding of all those concepts is still very fresh so I might say stupid things…

Ignoring gamut compression, I expect from the DRT:

1. Luminance to be compressed from scene values (~unbounded) to display values (bounded)
2. Hue to be preserved
3. Relative lightness to be preserved across pixels (if, in scene-space, object A is perceived as lighter/brighter than object B, that relationship should still hold in display-space)

1. is the main point of tone mapping, 2. is AFAIU the main point of ACESNext’s DRT and 3. is something I haven’t seen anywhere so far but that I find very interesting.

I believe OpenDRT and JzDT do not satisfy 3.

I suspect 3. might be an interesting constraint as it removes some degree of freedom from the space of possible solutions.

Most notably, I believe it makes the path-to-white an inevitable consequence of the constraints: it does not need to be “engineered” nor needs to be parameterizable.

Here’s how to construct the algorithm:

  • To satisfy 1. and 3., the tone curve needs to be applied to the lightness (Jz if we use JzAzBz as our CAM).

  • If we keep the chromaticity values constant (Az and Bz), we can deduce some corresponding display-referred RGB values.
    Assuming we choose correctly the output range of our tone curve, those display-referred RGB values can be all made to be within their respective possible range (ie. no clamping).
    All constraints are satisfied and this algorithm is also chroma-preserving (was not a goal though) but this leads to an issue:
    The brightest displayed white will only be able to be as bright as the dimmest fully saturated primary color of the display (ie. primary blue).
    This would lead to images generally dimmer than one would expect from their display hardware.

  • We can introduce a new constraint:
    4. Output colors must be able to span the full range of colors of the display-referred colorspace.

  • To get back the full range of luminance of the display-referred space, we need to expand the tone scale output range accordingly but then some overly bright and pure scene-referred colors will get outside of their display-referred colorspace.

  • The solution is to allow the chroma (computed by converting Az and Bz to chroma and hue) to be variable and scale down exactly by as much as needed to satisfy all the other constraints (incl. 4.).

I haven’t put this algorithm to code but I believe that once a CAM and a tone curve are chosen, the rest of the implementation should be pretty tightly defined.

I guess that solving the equation to find the value of the chroma might not be trivial within the framework of non-linear color spaces like JzAzBz but it should be doable.

To conclude, I believe that

  • 3. is a nice and intuitive property

  • that it is useful to explicitly state 4.

  • they can both lead to a tightly defined algorithm with a native and unparameterized path-to-white.

PS1: I guess that maybe the hue will be preserved in a more perception-accurate way because it is managed in the JzAzBz space instead of the LMS space.

PS2: Another consequence is that the tone function is not applied anymore in linear LMS space but in non-linear JzAzBz space. I guess that means applying directly an adapted S-curve without embedding it in the Michaelis-Menten model since JzAzBz’s PQ function is already doing a similar job.

4 Likes

Hello Paul and welcome to ACESCentral !

This is exactly what I have been through as well… You’re in the right place ! :wink:

This is not the first time I hear this and I think some people on this forum would be especially interested in this part.

This is a really interesting point of view. Thanks for sharing it ! Maybe it could be discussed during next meeting… I particularly like how you’re deconstructing the mechanics point by point.

Chris

1 Like

JzDT is much better at this than OpenDRT. From my art technical director perspective, it maintains lighting relationships much better than OpenDRT does (esp. w.r.t. brightly lit blues) although it does not produce as nice colours. I would agree with him. Once I have all the comments from my team in their final form, I will post them here.

Hi Paul,

Interesting post. Great to read and think about new ideas and approaches.

A few comments on first digestion:

Lightness is a tricky scale to predict, and I would not try to base a requirement on lightness.
An excellent summary paper about the difficulties with lightness:
https://www.sciencedirect.com/science/article/pii/S0960982299002493

But basically, your intuition about the monotonousness of the transform is not wrong; just the dimensions need to be discussed.

Also, I am not sure how you deduce:

[quote=“paulgdpr, post:14, topic:4008”]
Assuming we choose correctly the output range of our tone curve, those display-referred RGB values can be all made to be within their respective possible range (ie. no clamping).

[quote=“paulgdpr, post:14, topic:4008”]

while maintaining the (~unbounded) condition:

Further, I am wondering about:

I think you cannot easily directly relate from a CAM tone mapping to display units (this is one fundamental issue with CAM inspired spaces). Display Gamuts are oddly shaped in there.
This needs a bit of thoughts I guess, and this also applies the the rest of the algorithm you propose.

You also forsee this here, I think:

And I would like to mention that one crucial attribute for the DRT is “simplicity” and that the algorithm needs to be GPU friendly.

This all depends on the definition of “hue”.
There is some degree of freedom in designing your experiments and models and then fitting the models to the experimental results.

Be aware that PQ was fitted to JNDs (just noticeable differences), so encoding images with the minimal possible consumption of bits is perfect in PQ. This is what its design goals were, I guess. And also where it shines.
However, larger-scale equal distances is an entirely different perception and might not agree with JNDs.

I hope some of this helps.
Daniele

6 Likes

So if it works fast on a GPU and looks like ass, this is a design win?

I cannot believe that this sort of metric should hold any weight whatsoever in the act of image formation. Solve the problem of image formation first, worry about the performance and compute implications second. Having priorities this distorted is shocking.

L* is essentially an observer sensation metric derived from linear observer stimulus^1, which can be taken back to display colourimetry. The path from an observer sensation appearance metric back to currents and displays is perhaps feasible.

1 - Yes we can all acknowledge the fundamental brokenness of datasets derived from flicker photometry. Just using it as an example that we can indeed have a potential fluidity between observer sensation and observer stimulus that would allow us to get back to display colourimetry.

I haven’t read that in what @daniele said, he only highlighted that simplicity should be a critical requirement which is reasonable given it prevents the creation of steam factory. He never said it should take precedence on everything else.

Any research on this particular topic?

Of course I appreciate and understand the general intention of what Daniele said here. There are some problems with uttering the word “simplicity” however, much like “beautiful”; it’s just a seductive word.

At some point, a model will either sufficiently model what it is attempting to achieve, and fail at other potential things. My point is that to some degree, “simplicity” is just a meaningless word. How many steps do we need to take colourimetry from one display to another? Why? What is “simpler” here?

The critical thing here is a reevaluation of the process of image formation. It’s as much as brainstorming session as anything. I’d hope that such an attempt wasn’t hindered by chasing mythical dragons and leprechauns.

There’s an absolute proverbial metric ton of interesting stuff out there that laser focuses on aspects. I appreciate Lennie / Pokorny / Smith Luminance for a decent wide lens summation of the surface. 1931’s busted up aspect is nicely stated in Stockman’s Cone Fundamentals and CIE Standards page 90.

I’m interested by citations that state that the datasets obtained by flicker photometry are broken. You link Stockman but the cone fundamentals are tightly coupled to the CMFs by a mere linear relationship, if the datasets were so fundamentally broken as you imply it, you would not be able to do that at all.

The goal of flicker photometry is to frequently derive “equivalence” in brightness / relative lightness of chromatic stimulus.

Under that basic definition, if the testing does not actually model what it is attempting to measure, I would say that’s pretty broken. See how almost every CAM fails miserably at this basic task.

Luminance as a metric of observer stimulus response connected to observer sensation, is only more accurate as it approaches achromatic. Seems to be a bit of a tautological problem.

1 Like

I still don’t see any citation, just an opinion (which is fine in itself) :slight_smile:

The purpose of heterochromatic flicker photometry is to measure the spectral sensitivity of the HVS. The datasets produces by this method are verified by other approaches, for example electro-retinograms as shown by Aiba, Alpern and Maaseidvaag (1967):

If one take two radiometrically matching colours stimulus, and scale them so that they photometrically match, they will certainly appear much closer in apparent brightness, but not necessarily perceived to match, which, indeed, hints at some more complex mechanisms happening in the eye and cortex.

Does it means that the datasets broken? Certainly not! It means that we are far from having all the answers. Heterochromatic brightness matching experiments show that it is complex business, e.g. CIE 1988. To quote Conway et al. (2017):

We can now predict, with reasonable precision, how the three classes of cones react to any given physical stimulus. Yet many mysteries about how that code leads to the perception of color remain.

I was under the impression that Frederic Ives invented the method to deal with the subject of this discussion; brightness matching, and derive luminous efficacy from.

“Spectral sensitivity” could be considered to have been derived a number of ways, not limited to even the Wright and Guild split field experiments. Fair?

At any rate, specifically on the subject of equivalent chromatic brightness, flicker photometry bombs out. I’m not sure why someone would have an issue with that statement, but here we are.

Again, flicker photometry fails here, or in the most generous interpretation, only relates to a partial component at work.

The point I was trying to make was that, at least as far as L*’s relationship to (bogus flicker frequency derived) luminance, is that it is at least feasible to be able to move fluidly back and forth between an observer sensation appearance model and display colourimetry.

As pointed out (and you just confirmed in the above quote), the heterochromatic flicker photometry derived datasets, are verified by many types of experimentation and they produce, for practical purposes, the spectral sensitivity of the HVS, i.e. sensitivity to electro-magnetic as a function of wavelength. It is one of the basis on which the entire field of basic colorimetry is built on and allows for example, among many things, display calibration.

The failure of “knowledge of the spectral sensitivity of the HVS only” to explain our vision is a given. No one is contesting that, quite the opposite and this was recognised many decades ago, nothing really new here. Advanced colorimetry would not exist if everything was explained by strict sensitivity to radiant energy and we would not have that conversation in the first place.

Almost all of our current models are not even accounting for spatial induction, one simply has to look at this image to appreciate how gargantuan the task is:

“Illusory Red Thunder”
The electric locomotive appears to be reddish, though the pixel are not.
Copyright Akiyoshi Kitaoka 2020 (January 13)

What I’m contesting however are your claims here (or Twitter) that the datasets themselves are fundamentally broken while colour matching experiments, heterochromatic flicker photometry and ERG produce agreeing data.

I’m certainly willing to change my opinion on that but it needs to come with sourced research.

From Cinematic Color 2, quoting quotes :slight_smile:

Fairchild (2013) references a citation from Wyszecki (1973) describing basic colorimetry first:

Colorimetry, in its strict sense, is a tool used to making a prediction on whether two lights (visual stimuli) of different spectral power distributions will match in color for certain given conditions of observation. The prediction is made by determining the tristimulus values of the two visual stimuli. If the tristimulus values of a stimulus are identical to those of the other stimulus, a color match will be observed by an average observer with normal color vision.

And then advanced colorimetry:

Colorimetry in its broader sense includes methods of assessing the appearance of color stimuli presented to the observer in complicated surroundings as they may occur in everyday life. This is considered the ultimate goal of colorimetry, but because of its enormous complexity, this goal is far from being reached. On the other hand, certain more restricted aspects of the overall problem of predicting color appearance of stimuli seem somewhat less elusive. The outstanding examples are the measurement of color differences, whiteness, and chromatic adaptation. Though these problems are still essentially unresolved, the developments in these areas are of considerable interest and practical importance.

Couldn’t disagree more. Very, very few folks discuss g0, Macadam moments, Evans etc. in the discussion of image formation. But if that’s your view, so be it.

Why obfuscate something with additional complexities when more basic and profound issues are missing in our discussions of image formation?

Escaping values becoming device dependent and the complete bed poop of tonality theory and application in our current options on image formation strike me as far larger low hanging fruit. Hence why focusing on higher priority items as @paulgdpr has hinted seems far more rewarding that worrying about layers of nuance on top.

The basics idea of measuring brightness of chroma mixtures does not, in any way, work with flicker photometry. End. Nothing more to discuss.

If the measurement and resulting metric does not work, it isn’t worth talking about as “valid”.

Someone might want to say they are measuring human intellect using a cloth tape measure and that their measurement of human heads are consistent. The veracity of the claim is what is at issue.

I’ve spent a heck of a lot of time drilling into this subject and the only conclusion I can arrive at is that flicker photometry and resulting luminance is a misshapen bowl of, if we are generous, half baked nonsense with respect to building atop of with specific attention to the subject of relative brightness / lightness at hand in this thread. Such efforts must go further to potentially crack the specific nut Paul appears to be citing. As is potentially verifiable with some experimental testing on imagery.

Is there some potential nuance in the discussion for binding the need against known luminance? Absolutely! But then let’s make that part of the discussion.

If you would like to keep banging that drum, please explain what is gained. I’ve outlined the problems with flicker photometry based models and metrics as best as I am able, and I believe the issues are extremely pertinent to this specific discussion.

I can’t see how belaboring a rather rehashed point is in the service of resolving some critical issues in image formation. Again, specifically addressing the subject matter here.

The rest of us will try to move forward.

You are, again, c.f. Daniele above, putting words in someone else mouth. I said that the problems you are referring to are well known and have been documented for decades. As you reference Evans, he did spend a lot of time on those aspects, not last year or last decade but from the end of the 50s, i.e. ~60 years ago!

Because it is as soon as stimuli are not seen in isolation anymore that basic colorimetry starts to tank. Appropriately staying with Evans for a little more, I unfortunately don’t have The Perception of Color but there is a review from Heckaman and Fairchild (2004) that has a relevant quote:

Evans sums up the central thesis of his book. The fact that all colors can be matched by mixtures of three others and the establishment of three psychophysical attributes for such a match, the assumption since the time of Hemholtz has been that there must only be three perceptual attributes of color. In Evans’ own words, “ … the three-dimensional requirement for perception is logical only if it is also assumed that the appearance of the stimulus is controlled entirely by the stimulus itself. And this is true only when the stimulus is itself the only thing affecting the eye, that is, the isolated stimulus case.”

As we are somehow doing circles here, and given your experience on the subject, you should be able to produce a clear and sourced write-up demonstrating how the datasets are fundamentally broken, proposal on how to fix them and publish it on a peer-reviewed journal (or pre-pub like arXiv). Until this is done and your publication can been reviewed, it will be very hard to engage with you on this topic because there is very little substance to talk about.

1 Like

In colour science. Please show me this furious discussion around generalized brightness in image formation? I also have been following the threads for a while and I promise you that very few folks were even chasing MacAdam moments, Sinden, Evans, and HKE around here?

Why on earth would anyone care to do this when the results are glaringly obvious?

How on earth does wasting one’s time on writing papers on a subject that can be visibly demonstrated in a DCC in five minutes help chase image formation ideas? Or is the point to spin wheels looping over rubbish that doesn’t matter that is clearly self evident since Sinden in 1920?