Thanks again for your continued work on this Nick, this appears to be one of the most critical steps in the whole transform, so any and all tweaks need to be tried.
The part that hurt my head the most with the P3 clipping and the 709 not clipping the green was not that the different gamuts gave different results, it was that 709 Limited P3 gave different results in P3!
The idea to blend the two images was not immediately useful to fix the problem, but it suggested that an intermediate abstract gamut might be a useful idea.
From the perspective of 709 limited P3, the 709 is just an abstract gamut, so maybe another “ideal” abstract gamut could offer the desired properties.
here are two examples, roughly based on 709 near “Arri Bar green” (P3 cusp is not accurate, but could represent any larger display gamut), and the sample is roughly “Arri Bar green” values for reference.
The first is very simple, close to an average, but it specifically has a cusp with a higher J value than 709.
The second one was done with a simple bezier (not sure if that is too complex for current options), but something like that may help with smoothness.
Only suggesting any of this because the gamut mapper appears sensitive to the cusp position and the results from P3 were so unintuitive compared to 709.
Probably not a solution, given the requirements to map to any display, but the idea has come up a few times so it might be a useful experiment to see what an adjusted “virtual cusp” might have on the output.
At the very least, It may help determine the ideal behaviour of the gamut mapper.