Output Transform Tone Scale

[jedsmith]

In an effort to tackle a narrow portion of the complex topic this working group is engaged with, I would like to discuss here the Tonescale.

The way I see it, the Output Transform should consist of a few different modules:
1). Tonescale: Compress dynamic range from scene-linear to display linear.
2). Gamut Mapping: Map hue from source gamut to display gamut. Perhaps there is some default rendering decisions in here as well such as desaturation of highlights, shadows, etc.
3). Display EOTF: Apply the display transfer function. Perhaps there are other things in here as well such as surround compensation.

In the current Output Transform, 1 and 2 are intertwined because the tonescale is applied directly to the RGB channels. (Not to mention the 2 saturation matrices applied before and after the tonescale). This is problematic because hue is not exposure invariant (brighter blues turn purple). It also adds complexity.

Inspired by Doug Walker’s excellent presentation: (SMPTE 2017: Core Color Rendering Algorithms For High Dynamic Range Display), I’ve put together a Nuke setup to apply the Single Stage Tone Scale algorithm as a ratio preserving tone curve.

I’ve included options for different norms, so that we compare the differences in how the norm affects the image rendering.

• Luminance - Rec.709 luminance weighting.
• Max RGB - Calculates norm with max(r,g,b)
• Weighted Power - Uses the weighted power norm described in Doug Walker’s talk. Options for basic 1.0 weights, and for the weighted yellow power norm coefficients.

Here are some images.

There is no gamut mapping applied here. So it’s ACEScg directly to display.

Because the tonescale is hue invariant, it’s possible that gamut conversion and hue alterations could be done before the tonescale in the scene-referred domain, or after the tonescale in the display-referred domain.

Another concern would be how to make saturated highlights look bright again. This process renders saturated bright highlights with a very dim appearance. How could we control saturation of highlights (and maybe shadows too) in a simple way?

A ratio preserving tone curves gives us a lot more control, but we may need to develop some methods to recreate the aspects of the previous rendering that we have lost with it.

[Troy_James_Sobotka (troy_s)]

Amazing work Jed!

@jedsmith

1). Tonescale: Compress dynamic range from scene-linear to display linear.
2). Gamut Mapping: Map hue from source gamut to display gamut. Perhaps there is some default rendering decisions in here as well such as desaturation of highlights, shadows, etc.

I preface this series of questions with the fact that I have no idea what anyone means by “tone”, let alone “mapping”. I’m sure there are people facepalming.

1. I’d dearly appreciate it if someone could explain what the X input axis is and the Y output axis is?
   1.a How is this relationship related to “tone”?

@jedsmith

Another concern would be how to make saturated highlights look bright again. This process renders saturated bright highlights with a very dim appearance.

2. Isn’t the point of an aesthetic transfer function to perform a compression of an input gamut volume along X to an output gamut volume along Y?
   2.a In this case, isn’t the “dimming” exactly as expected?
   2.b If this is expected, is there an error of logic with respect to the dimension of the input X range?
   2.c Is there a limit to the display gamut volume range of expression of output Y?
   2.c.1 If there is a potential limit, is there a means to glean a maximal quantization increment with respect to input X and output Y?
   2.d Are there quantization limits at both the upper and lower end impacting the expression of a given radiometric-like chromaticity and / or perceptual hue?
3. With respect to norms, how does this relate to the aesthetic ground-truth of the past century of photographic media?
   3.a Wasn’t film explicitly radiometric electromagnetic energy transformed into chemical energy?
   3.b What happens with high energy radiometric-like saturated mixtures with respect to high energy, lower saturated mixtures?
   3.c If there is a difference in 3.b, does this create a cognitive dissonance with respect to the typical aesthetic ground-truth of the past hundred years of photographic media?

Again, tremendous work.

PS:

1. Assuming the luminance is used as the input lookup value, it would seem the magenta skew might indicate that the decompression is being applied incorrectly in the luminance norm case?
2. It might be informative to see how non-maximally saturated values translate to each approach?

[jedsmith]

Side note: I’ll continue to use the term “tonescale” in this post, because it is what is used in the ACES CTL. I remember using the word tone to refer to sections of the exposure range back in college developing black and white film negative in the darkroom. I do agree the term tone is a bit ambiguous though. If there is a more appropriate term to refer to the process of compressing scene intensity to display intensity, please let me know.

A Decision Taken For Granted

Implicit in my previous post is the assumption that we had collectively decided to use a chromaticity-preserving “tone scale”. Maybe that’s not the case, or maybe we haven’t fully evaluated the implications of the decision.

For me it seems that this is the first decision that needs to be made, because every step after follows from this.

The reason to use a chromaticity-preserving tonescale is clear to me. Input chromaticity matches output chromaticity. Here are some pictures to show visually what that means.

Here’s an image which shows what I mean pretty well. @ChrisBrejon’s cg_sRGB_spheres_001_aces.exr.
Shown above is the image in acescg rendered directly into a jpg and uploaded here. No tonescale applied.

And here’s the same image using the Weighted Yellow Power Norm tonescale. This is using the ACES SSTS algorithm. Note this is display linear, no display EOTF applied.

Next here is the same image with RGB per-channel tonemapping.

And now for some plots of the same images in the same order, in CIE 1931 Yxy chromaticity space.

It is clear that a norm-based tonescale does not alter the chromaticity of the input image. And that an RGB based tonescale is doing all sorts of things, some nice, some not so nice.

Looking at the same image from a roughly 45 degree angle, we can see what is happening to saturation over the range of brightness. In the RGB per-channel approach, saturation approaches 0 as brightness approaches the maximum code value or 1.0 in display-referred. The problem is that we have no control over the way that this happens.

From Which Place Do We Start

If we were starting to build a display rendering transform, I personally would much rather start from the result of the norm-based approach, because I have control over what is happening. If we started from the per-channel approach, sure there are some nice looking results, but there are also some not-so-nice looking byproducts which would be very difficult to reverse later on.

If we all agree that starting from a chromaticity-preserving tonescale is the desired method, then we could move on to talking about how to emulate a look. For example, what are the aesthetic characteristics that we would like to preserve, and how do we make that happen.

I’m going to make another thread about gamut mapping and throw out some ideas for some things.

A Question: Proper Handling of EOTF?

One question I do have for some of the smarter minds than my own is should the display EOTF be included in the chromaticity-preserving tonescale? That is, should the display EOTF desaturate and change chromaticity values when it is applied, for example if a gamma 2.4 transform is applied for Rec.709, or should compensation for this desaturation happen before?

I’m getting rather nice results with an order of operations that looks like this:

Gamut compression -> Tonescale to display linear -> 3x3 Matrix to convert to display primaries -> Display EOTF

But I would be curious to hear thoughts on this.

Edit: Here’s the nuke script used to generate these images.20190119_tonescales.nk (681.7 KB)

[jedsmith]

@Troy_James_Sobotka

Amazing work Jed!

Thanks for your post @Troy_James_Sobotka. Your questions helped me frame some of my thinking about this topic.

@Troy_James_Sobotka

Assuming the luminance is used as the input lookup value, it would seem the magenta skew might indicate that the decompression is being applied incorrectly in the luminance norm case?

Apologies, my image was a bit misleading for the Luminance Norm examples. There is no chromaticity shift with this approach, but the image after tonescale has values > 1.0 in the blue channel, which got clipped and caused an apparent magenta hue in the jpg.

After quite a bit of testing, I really like the behavior of the weighted power norm approach. Adjusting the weights allows good control over the brightness of different hue ranges without changing chromaticity, and would allow for another point where an aesthetic decision could be made.

[Troy_James_Sobotka (troy_s)]

@jedsmith

After quite a bit of testing, I really like the behavior of the weighted power norm approach.

The issues with all non-radiometric like norms is that they add a convolution on top of the curve. As in the meaning of the curve is not clear, because there is another energy “balancing” at work, which will distort the curve at various points, calling into question the whole idea of a single curve to begin with.

Wiser minds can probably plot this issue without breaking a sweat.

[Alexander_Forsythe (Alex Forsythe)]

Hey Jed … thanks for digging in …

While we certainly played with the maxRGB concept during the development of ACES 1.0, Doug’s Weighted Power algorithm came after.

Just for kicks, I applied it to some of the ACES 1.0 test images. Interestingly, it produced similar pictorial results to the maxRGB that caused us to move back to RGB tonescale methods in the current ACES system. rgbMax and weighted power certainly do a better job at preserving hue invariance as exposure increases, but as you noted, the saturated, bright highlights are certainly a less-than-desirable side-effect. We noted that highlights don’t desaturate until the source is clipped regardless how high their radiance is.

In the end we came to the conclusion that you hue linearity and saturation are desirable … unless they aren’t.

Each of the following images was rendered using @jedsmith nuke script referenced above using the weighted power option with yellow weights and the BT1886 EOTF

Retaining hue linearity and saturation looks good on neons and tail lights

Not so good on the sun.
Hard edged sun with colored rim typical of what we saw with maxRGB

Exposure sweep of MacBeth Color Checker

Exposure sweep of MacBeth Color Checker lifted 10 stops in ACES linear space (ACES * 2^{10}) - Nothing blows out unless the source data converges in all channels as happens with camera clipping.

[garrett.strudler]

@jedsmith

That is, should the display EOTF desaturate and change chromaticity values when it is applied, for example if a gamma 2.4 transform is applied for Rec.709, or should compensation for this desaturation happen before?

In my mind this is likely a question of if HDR and SDR (and DCI) will need different treatment of the highlight desaturation. A “universal” desaturation would clearly be ideal if it works across the board.

[garrett.strudler]

@Troy_James_Sobotka

The issues with all non-radiometric like norms is that they add a convolution on top of the curve. As in the meaning of the curve is not clear, because there is another energy “balancing” at work, which will distort the curve at various points, calling into question the whole idea of a single curve to begin with.

How “convoluted” are we talking? Invertibility seems high on the priority list, but I don’t think that necessarily restricts us to a simple curve if it can reasonably be undone (although I infer from your post that you think we’re beyond that in this model).

[jedsmith]

@jedsmith

should the display EOTF desaturate and change chromaticity values when it is applied

This was a bit of a brain-fart stupid question on my part, which was quickly clarified by multiple people in the last meeting.

To summarize: The display EOTF transform should not preserve chromaticity values. The EOTF applies a temporary encoding as the image is sent “across the wire” to the display. At the display, the opposite of the EOTF is applied before the picture is rendered on the screen. In fact, as @doug_walker pointed out in the meeting, the display EOTF should probably more accurately be referred to as the “Inverse EOTF”, because it un-does the transform which the display later applies.

Long story-short, if the “Inverse EOTF” is applied properly, the radiometric display-linear code values just previous to the “Inverse EOTF” transform will be displayed exactly the same as light emitted from the monitor.

@garrett.strudler

In my mind this is likely a question of if HDR and SDR (and DCI) will need different treatment of the highlight desaturation. A “universal” desaturation would clearly be ideal if it works across the board.

Agreed. Desaturation should be a function of the compression curve rather than a separate operation. Exactly how is an open question though

[Troy_James_Sobotka (troy_s)]

@jedsmith

Desaturation should be a function of the compression curve

Except it is a different medium to film, no? What does a “tone” mean here? Is it somehow different to the double duty that a density plot represents with film?

@garrett.strudler

How “convoluted” are we talking?

It’s a shifting nonlinear result underneath a nonlinear curve. This would seem to make it nearly impossible to comprehend the meaning of the curve in question, given the plot is nonlinear that it is on top of?

[Thomas_Mansencal]

@jedsmith

This was a bit of a brain-fart stupid question on my part, which was quickly clarified by multiple people in the last meeting.

To summarize: The display EOTF transform should not preserve chromaticity values. The EOTF applies a temporary encoding as the image is sent “across the wire” to the display. At the display, the opposite of the EOTF is applied before the picture is rendered on the screen.

Yes, in the perfect world with no performance, energy or bandwidth constraints, the EOTF is not required, e.g. a full floating-point display chain. Generally best to leave it alone and think about it as a no-op even though it is not necessarily strictly the case. Put another way we do have an EOTF mostly for (good) cost reasons.

[ChrisBrejon (Christophe Brejon)]

Very interesting point. I always felt that something like this was going on but never dared to ask. Thanks for having clarified this topic !