I have been doing some further investigation into where these small round trip errors creep in, using @priikone’s v53_pex4. In a 17^3 unit cube, the colour with the largest round-trip error is [0.0, 0.9375, 0.0].
Using Nuke’s debugPrint() statements at the various steps though the forward and inverse process, I get the following results:
// inverse transform
float3 srcRGB = 0, 0.9375, 0
float3 luminanceRGB = 0, 85.6507, 0
float3 XYZ = 30.6274, 61.2547, 10.2091
float3 compressedJMh = 69.0652, 71.6736, 136.472
float3 tonemappedJMh = 70.0035, 84.1431, 136.472
float3 JMh = 101.084, 121.587, 136.472
float3 dstRGB = 0.550096, 1.99705, 0.0747953
// forward transform
float3 srcRGB = 0.550096, 1.99705, 0.0747953
float3 JMh = 101.084, 121.587, 136.472
float3 tonemappedJMh = 70.0035, 84.1429, 136.472
float3 compressedJMh = 69.0652, 71.6736, 136.472
float3 luminanceXYZ = 30.6274, 61.2547, 10.2091
float3 luminanceRGB = 0.000249863, 85.6506, 2.86102e-06
float3 dstRGB = 0.00456816, 0.9375, 0
As can be seen, everything round-trips perfectly as far as the output XYZ of the forward transform. It is in the final conversion to RGB that a small linear error creeps in, and then the BT.1886 gamma amplifies that.
So everything we do in JMh is inverting perfectly. It’s the final matrixing from XYZ to RGB which introduces the issue (such as it is).
The XYZ to RGB matrix is calculated in the code as the inverse of the RGB to XYZ matrix, so the problem does not come from declaring matrices at insufficient precision. Would performing the matrix operations at 1.0 scale instead of 100 scale help? I am not sufficiently familiar with the limitations of floating point maths to know.
Since we have talked about creating a transform which goes from linear AP0 to display-referred XYZ, and leaving the final RGB encoding up to implementers, it could be argued that we have a perfect round trip, with no precision issues! But that is just kicking the problem further down the line!