This is a first indicator of “wrong domain” . If you start to do stuff per hue it becomes questionable why you would go into that space anyway…
And what does “perceptually uniform” actually mean in this context?
3 Likes
I tried using a few different perceptually uniform colorspaces to do this: Oklab, IPT, ICtCp, JzAzBz - they all yield pretty similar results, just slightly shifting the behavior of the hue as it travels back to neutral.
I guess I started out thinking of this problem more as a gamut mapping problem, looking at @ChrisBrejon 's lightsaber render and other very saturated hues. Using a perceptually uniform space made sense in my color science neophyte brain in order to try to preserve “natural looking” hues as they desaturate. This thinking may be totally incorrect and not at all what a perceptually uniform colorspace is designed for.
Attempting to desaturate highlights (and shadows) as a component of a chromaticity preserving display rendering should probably be done in a much simpler way. As @daniele said, in a different domain.
I don’t think it is incorrect quite the opposite, but the way they model our perception is maybe not appropriate for our needs. Also, given their design simplicity, they cannot be good at predicting lightness, chroma and hue at same time. Their is so much that a 3x3 matrix and a power-like function can do! The other space that might be also interesting to look at is IgTgPg, even though it does not produce great gradients!
I will render some hue stripes later for all of them.
Don’t get me wrong, I think this is all great, but also keep in mind that those colour space were never designed for the task at hand.
Yes, this was my point! 
Disclaimer: The following is not an approach I would suggest be used for producing good looking images, but merely an interesting and informative exercise to better understand the problem at hand.
Back to Basics
The last several days I’ve been thinking about the highlight desaturation problem. With display rendering there are a lot of intertwined problem areas, and it’s very easy (at least for me) to get confused. Like earlier when I was thinking about gamut compression and highlight desaturation as one problem. They are two different problems. Related, but not the same.
So how can we simplify and isolate these problem areas? We could ignore “out of display gamut” issues. We could ignore tonemapping or intensity scaling issues.
NaiveDisplayTransform
In an effort to simplify and focus on highlight compression and desaturation, I made a nuke node called NaiveDisplayTransform. As the name suggests, it undertakes a very naive approach to mapping scene-linear to display-linear.
- Maps an input range in scene-linear from
linear_starttolinear_end, defined in stops above and below middle gray, to an output range in display linear from 0 todisplay linear end. - From
display linear endto 1.0, apply a compression curve to the scene linear values. The compression curve compresses infinity to a value oflimitstops abovelinear end. In other words: if limit is 1, all highlight values fromlinear endto infinity will be compressed into a range of 1 stop. The biggerlimitis, the more range there will be for the compressed highlight values. - Where highlight values are compressed, desaturate.
The simplest possible display transform would be a linear to linear mapping. We take a range of scene-linear values and remap them to a range of display-linear values. We apply the inverse EOTF. The scene linear values are displayed directly on the display, up until the values clip. For the range that can be displayed, (ignoring spectral emission differences, calibration, and other complications) there should be a 1 to 1 correspondence between scene luminance and display luminance.
Since pictures are easier to understand than words (at least for me), here’s a video demonstration of the concept and the tool.
When Channels Clip
With a simple 1 to 1 mapping, we can focus on how highlight values behave as they approach 100% display emission.
With no highlight compression and no desaturation, hue shifts are introduced for non achromatic colors, because as one component clips, the other components continue to increase. So we need some way to “handle” these components as they increase in brightness, to remove hue shifts as the components approach clipping. To do this we can move the components toward the achromatic axis as their luminance increases.
Here’s a video demo with some interesting plots of what’s going on.
Technical Approach
Amidst an extended conversation with @Troy_James_Sobotka (thanks for your patience with my stupid questions), I got to thinking about the Inverse RGB Ratios approach that we ended up using in the Gamut Mapping VWG. In that approach we were using the inverse rgb ratios to push out of gamut values back towards the achromatic axis.
inverse_rgb_ratio = max(r,g,b) - rgb
If we instead used a constant value that we wanted to “collapse” our components towards, we could do
lin_max = 4.0
inverse_rgb_ratio = lin_max - rgb
If we add inverse_rgb_ratio to rgb, we will get lin_max everywhere. But if we modulate inverse_rgb_ratio by some factor which describes how much we want to desaturate, then we get a simple and effective method of highlight desaturation.
The best way I’ve found so far (and I think there’s better approaches), is to modulate the inverse_rgb_ratio by the complement of the compression factor. When we use the norm to do the highlight compression, we do
norm = max(r,g,b)
intensity_scale = compress(norm) / norm
scaled_rgb = intensity_scale * rgb
compression_factor = 1 - intensity_scale
intensity_scale here is kindof like the derivative of the curve: it represents how much the compress function is altering the norm.
Then we can do
lin_max = 4.0
inverse_rgb_ratio = lin_max - c
desaturation_factor = inverse_rgb_ratio * compression_factor
desat_rgb = rgb + desaturation_factor
Here’s a video walkthrough of the technical approach.
Considering how stupid and simple this approach is it actually doesn’t look half bad in my aesthetic opinion.
Here’s a video of how the transform looks on some example test images.
And finally here’s the NaiveDisplayTransform node if you want to play around with it. No blinkscript so it should work fine in Nuke Non-Commercial.
EDIT - since the above link points to a gist that has been updated, here is the original nuke script as it existed at the time of this post: NaiveDisplayTransform_v01.nk (18.3 KB)
Next step for me is seeing how this same approach might be applied with a more traditional sigmoidal curve for input range compression.
Just wanted to share what I’ve been up to in case it’s useful for anyone here.
4 Likes
Nice stuff !
I have been playing with the Nuke script this week and it has given me interesting results. I think it is great to go back to basics in a way (at least for me !) in order to fully understand what we are dealing with.
I have done a quick image of the tool’s options/parameters if anyone is interested. It helped me to grasp better these concepts.
I am looking forward to future updates of the tool. Let me know if my “mockup” is wrong or incorrect, I’d be happy to update it.
Chris
1 Like
Thanks @ChrisBrejon! Now that you’ve made this excellent diagram explaining everything, I reworked the parameters a bit to hopefully make it more clear what is going on. I realized after reading my post again with a fresh brain that there were some things that could be made more clear and simplified.
I reworked the parameters so that the naming is more consistent, and so that the linear value that is calculated is displayed right below the value you are adjusting.
Everything functions the same way except for the limit parameter, which I’ve changed the name of to compression and altered how it works. Now instead of mapping infinity to the value that you specify in stops above lin_white, you specify a value in stops above lin_white, and that value is compressed to display maximum. In other words, compression calculates the max value, which is the maximum scene-linear value that will be represented on the display.
I’ve also exposed the strength parameter, so you can adjust the slope of the compression curve. It seems like you need control over this to get the best results, depending on the dwhite and compression value you have set.
As before here’s a little video demo of the updates and changes I’ve made.
EDIT - Here is the nuke script described in the above screenshot
NaiveDisplayTransform_v02.nk (6.9 KB)
I also made some further usability improvements and additional parameters, which I did not make a post about. Here is that updated version:
NaiveDisplayTransform_v03.nk (7.6 KB)
EOTF should be EOTF^{-1} or Inverse\ EOTF and we should try to remove gamma if possible too!
3 Likes
Thanks @Thomas_Mansencal ! Indeed, that’d be much better this way ! I’ll update the sketch asap.
Cool @jedsmith ! I’ll give it a try next week ! Great update and video explanation. Thanks for sharing !
Jed,
Thanks so much for the work and thoughts. There’s a lot here to react to. I’ll just throw out a few points to consider.
-
I don’t know if I’d characterize highlight desaturation as a problem. Sometimes it’s the effect you want, and sometimes it isn’t.
-
I think there’s value in recognizing, either conceptually or in practice, that the rendering transform, at some point, should yield display colorimetry and turning that display colorimetry into a signal is a process that’s pretty straight forward and objective (e.g. convert to display primaries, encoded with inverse EOTF, etc). The only wrinkle is gamut mapping. We’ve talked about this We’ve talked about the separation of rendering and creation of the display encoding a bunch in the past and it’s fine to separate that out but it’s not really part of the rendering discussion per-say. We discussed the concept in this document. http://j.mp/TB-2014-013.
-
Displaying linear scene data on a display, within the limits of that display’s capability, is a helpful tool at times. With the ACES 1.0 work we did that a lot. I also suggested doing that during the gamut mapping work to help visualize the ACES data. Generally speaking, to make a reasonable reproduction on an output medium you’re going to need to compensate for viewing flare and the surround associated with the display environment (among other things). But those two usually end up increasing the slope of the tone scale. I’d highly recommend Chapter 5 of Digital Color Management: Encoding Solutions Second Edition which does the topic way more justice than I ever could.
3 Likes
Hey Alex, thanks for the thoughts!
Thank you! I’m in this to learn, and I appreciate the reference. I’ll take a look at this. As I mentioned in my disclaimer above, this experiment is purely to better understand the problem at hand, not something I am putting forward as something to be considered as an aesthetically pleasing display rendering.
My Apologies for the imprecise terminology. As I’ve mentioned in the past, I could more accurately be referred to as a “Color Pragmatist” than a “Color Scientist”.
The problem that I’m attempting to understand in my ramblings above could perhaps be more precisely expressed.
What I’m trying to understand is what happens to color when approaching and passing the “top end” or max luminance of a display-referred gamut boundary. That is, for an rgb triplet which contains a color that is too bright to be represented on a display device, what happens to hue when one channel clips to display maximum and other channels do not. These unnatural renderings of hue in the upper portions of the luminance range is why I think gamut reduction as luminance approaches display maximum is critically important to rendering a good looking picture, rather than an aesthetic preference. The example photos that you posted here show the issue pretty clearly.
Of course the implicit assumption in all of this is that we would want a chromaticity preserving tonescale rather than a per-channel rgb approach which applies this luminance-gamut limiting as a byproduct.
Can you elaborate on why gamut mapping should not be a part of the rendering discussion? Surely there should be some type of handling for colorimetry that can not be reproduced on the display device in the default display rendering transform? I’m sure it’s something silly I didn’t think of, but like I said I’m in this to learn 
2 Likes
Hi Jed,
can you post again a link to your latest version of your Nuke Gizmo? Thanks.
Totally understand … thank you for playing with this and sharing what you’ve learned.
I think that’s the part I’m reacting to really. I don’t know if we do or don’t. A chromaticity preserving tone scale sounds great, makes great plots, looks great on some images and absolutely horrible on others. We spent a ton of time in the ACES 1.0 development process looking at a chromaticity preserving tone scale it it was like an onion. We’d peel back layers and just run into more and more issues. The models kept getting more complex and eventually we pulled the plug. I think it’s worth looking at again but I’m not sure we want to make it a hard requirement.
This is just the way my brain splits up the problem but going from scene-referred colorimetry to the colorimetry as we want the image to appear on the display is the rendering. For the sake of argument let’s call that output XYZ. There’s a separate, and relatively easy, task that needs to occur to take those output XYZ values and turn them into a set of code values. When those code values drive the display in question measurements of the light coming from the display should match the rendering output XYZ values.
Obviously, depending on the details of the rendering, the output XYZ values may exceed the capabilities of what the display can actual show. If that happens choices need to be made in the signal generation stage. With this sort of conceptual split though, you can do interesting things. For instance, you can make the rendering create output XYZ values for a limited dynamic range and a small set of primaries (e.g. rec709) and then generate the corresponding code values to correctly display those output XYZ values on a display with much higher dynamic range and a larger set of primaries. This effectively lets you use an HDR monitor as a SDR monitor without changing the monitor calibration. @sdyer and I do this all the time with our BVM-X300.
1 Like
The simple fact it would be such a departure to the current rendering makes it a no-go as a default for me. It does not mean we should not provide a way to be able to do it though, the advantage is that it could be introduced alongside and let people test and migrate their workflows to a hue-preserving one.
100% agreeing for the gamut-mapping point!
Cheers,
Thomas
Just to complement the hue-preserving discussion, I remembered the presentation Alex Fry (EA) gave at the GDC 2017: https://www.gdcvault.com/play/1024466/High-Dynamic-Range-Color-Grading
2 Likes
Sorry I may not have respond to this directly. I wasn’t trying to say gamut mapping wasn’t part of the rendering but rather the conversion from output Colorimetry to display code values. If the rendering produces values outside the displays capability obviously gamut mapping is needed. Probably better to make the rendering not produce out of gamut values to begin with though.
Thanks for clarifying. As is so common in conversations about color, differently understood terminology seems to be the source of my misunderstanding here. As your diagram shows, I believe you are thinking about “Rendering” as a separate step from the transform to get to display colorimetry, where I was thinking of it as the whole process to get from scene-linear to display.
I do wonder though - is it really possible to separate rendering from transformation to display colorimetry?
Isn’t this impossible? - (This may be another stupid question - like I said, color pragmatist)
What if we need to create a rendering for Rec.2020 and for Rec.709? Are we going to use the same gamut mapping approach? Would we limit the colors to the smaller one? Or would there be two different approaches specialized for each display gamut?
Certainly we could conceptually move the gamut mapping problem into a box between “Output XYZ” and “Display Code Values” but it seems to me like this would still need to be a specialized process for each output display gamut?
Please let me know if I’m missing something here, I am genuinely curious.
There has been a very interesting conversation on slack and I thought it would be worth doing a summary on acescentral.
my only point is that removing hue shifts as a default is not desirable because of the arguments Alex Forsythe brought up and the simple fact it would change the look massively. […] Reminded me Alex Fry GDC presentation from 2017 where he explained that he was adamant about removing them but it did not worked for all cases, quite the opposite!
Which @Troy_James_Sobotka replied :
Optionally, I think the point is being missed.
- The display can’t represent the colour of the flame, so it skews.
- It is possible, if one were so inclined, to add a saturation adjustment to return more of the “yellowish” flame.
- It is impossible to remove the saturation surface of the image once the skews are introduced.
[…] But again, the crux of this discussion is essentially about why bother with colour science at all if it never comes out of the display?
The conservation followed like this with @Thomas_Mansencal explaining :
Without entering into subjective discussions, changing something like the default tonescale contrast has much less impact than making the tonescale hue preserving. So making the latter change a default in a new ACES rendering transform has much deeper consequences, so I don’t think it is a good idea, it should be optional. If you have built a ton of assets under the current rendering transform and you end up in a situation where the new one forces you to rebalance all of them, it is not a great change. […] Not saying there is no problem, just saying that the medecine should not be the nuclear option.
@daniele then added :
If the skew is part of the Display Rendering it needs to do the same skew for all deliverable. In fact your cg assets will look completely different in HDR under current transforms. […] Well the starting point should be same appearance , I guess.
Which Thomas replied :
They will (look different) and it is acknowledged, entirely depends on the deliverables. And the client might also either be fine with a different look on HDR but even request for it. […] Just look at the Haywire trailer by Steven Soderbergh… Not saying that I like it, but the client pays the bills here and he is certainly pushing for that look. I don’t know if the appearance should be the same with vastly different class of displays: are clients expecting that the appearance of a 100nits, 10000nits and the real world appearance are the same? It is a fundamental question to which I have no answers. Say I’m paying extra bucks to go to HDR cinema, what is the point to get there if things look the same than SDR exhibition ? That is the type of questions producers ask themselves.
Then Daniele explained :
I am not saying HDR and SDR should be identical. But a yellow candle in SDR should not be orange in HDR I guess. At least I know many colourists which are confused when that happens.
To which Thomas replied :
I certainly could see that happening but all your pyro effects turning red/white under a hue-preserving rendering transform is equally bad, and it does not affect a single person but hundreds of people. All of sudden, all your Lookdev, FX, Lighting & Rendering teams are affected. Sometimes, broken by force of adoption is what is correct, but we are entering philosophical discussions here. And again, what feels broken for you might be a feature or the expected behaviour for others. […] I can certainly point at the dangers of changing the rendering transform in such a dramatic way that it preserves hues. […] Changes to a a large system like ACES must be incremental, you just can’t modify something as critical as the RRT without triggering a storm, instead you should give people the option to change their workflow, at their will and gradually. Happy to be corrected but it seems like the only sane thing to do to avoid having an angry mob knocking at the Academy’s door!
Jed entered the conversation :
2.0 seems like a good version to make a change to the default display rendering transform. I’m not sure there would be an angry mob if it looked better than the current version. I think a lot there are a lot of people especially in vfx who use aces because it is easy and off the shelf, who think that it is a “neutral” view transform, and are trusting it to represent their asset work faithfully, and they do not realise what is happening to the color that they are so careful to create faithfully once it goes through the view transform to the display. Certainly a chromaticity preserving rendering method is not the easiest path forward, but I do not think an approach should be discarded because it is difficult or inconvenient. As professional image makers we should be able to make intentional decisions about the appearance of color, to have control over what is happening. I think there is benefit in the approach of trying to pull apart the reasons why certain things are happening, why we are seeing certain artifacts. It is very difficult to break down problems like this because there are so many factors at play. But I think if we try to isolate problem areas maybe there are solutions we haven’t thought of.
And Daniele suggested :
You can make a LMT to match ACES 1.0?
@nick also shared his thoughts :
If we offer a few options out of the box, then I think the default should feel close to what we have now, to keep the people who already use ACES comfortable. But we could have a hue preserving alternate, and maybe people would gradually move over. Then we can deprecate it in 3.0.
A big question brough up by Thomas and Troy was about objectivity :
The problem is that looking better is highly subjective and any large DRT change has potentially a large cost.
I can’t help but find this “highly subjective” line of discourse completely quite disturbing. What’s our ground truth? We have a working space. We have values. We are not displaying them. At all. Not even remotely.
Interestingly enough it looks like the famous K1S1 DRT is not hue-preserving.
Then @sdyer entered the conversation :
[…] all options are on the table - if they meet the requirements. Which we are trying to define. everyone here seems to be acting like decisions have already been made. Can we focus on defining absolute requirements and then maybe even things that “would be nice to have” (if we can get them without breaking other stuff)?
We then agreed that we would need to see examples of chromaticity preserving tone scale sounds great, makes great plots, looks great on some images and absolutely horrible on others to better understand what happened for the creation of ACES 1.0.
I also tried to give my tuppence :
Keeping skews for a major release doesn’t sound right to me. For a major release, we should not be shy of changing stuff. This is the moment to do things “right” or “better”… Not 3.0. By definition, a major change means stuff will change and eventually break.
Which Scott replied to :
I can tell you I am 100% in favor of changing stuff, even drastically, IF it fixes the things we consider broken and still delivers on the other things we want it to keep doing. The requirements define the solution. so figure out what it must do and then we can focus our efforts on making something that does. We can’t sell a big change jsut for the sake of changing things. But if we can show the many ways on which it improves known issues with v1, we will have a lot easier time convincing the stalwarts to change and the holdouts to revisit it.
Finally Daniele shared some thoughts :
I still have difficulties to accept the “cg assets” argument. If the assets are so fragile to the DRT in use how do they survive a normal grading session. All the cg and VFX elements I got my hands on work very well with different DRTs and looks. They look different, but so does all the live action too. […] In a way a cleaner DRT should actually help to produce neutral assets.
Final words from thomas about CG issues :
Not all assets are equally fragile, pyro elements certainly would be. You have a whole class of people using pre-rendered texture sheets and such in games. And they tweak the look through the DRT directly.
I have tried to reproduce this conversation in the most faithful way. If you think I haven’t done a good job, please let me know and I will modify/delete the post. But I thought it would be important to share with everyone on acescentral.
- Should the new Output Transforms be hue-preserving ?
- Should there be a default behaviour and then an optional one ?
- Do we want a smooth transition or a fresh start ?
- Should we consult (again) with studios/people using ACES to see what they think ?
Regards,
Chris
4 Likes