ACES Product Qualification

This is a followup post to some of the points introduced at last meeting. As we approach the next meeting, please continue thinking about some of the observations I raised and consider some of the discussion points. These are sort of stream-of-consciousness - feel free to respond to any that you feel comfortable commenting on. Or, if you feel more comfortable, just hold it for the meeting.

This week we will expand these fundamentals to see if they also include “What is an ACES show?”

Re: the existing Logo Program documents, including “Description and Qualification” document

Observations

  • The current documents are lengthy and steps numerous (although the current version of the documents is more concise than pre-2020).

  • Some current requirements are (or at least were) under-specified, especially those relating to AMF & CLF.

  • The existing “Major Features” seem like a good starting place for defining “tenets” of ACES:

    1. Produce scene-referred data
    2. Process color without compromising dynamic range or color gamut
    3. Display consistently across multiple display devices
    4. Enable on-set preview and look management
    5. Enable basic grade portability
    6. Enable color pipeline configuration portability
  • The enumerated Specific Features are useful, but likely not comprehensive.

  • Product categories seem like a good idea for providing examples, but there should probably be a way for hybrid categories or unexpected exceptions.

Discussion Prompts

  • What should it mean for a product to “support ACES”?
  • Which products need to support which features?
  • How can/should products be adequately qualified without the process of doing so being overly regimented?
    • There is a need to balance a thoroughness of qualification without being an overwhelming process that will discourage submissions
  • How can requirements be structured so as to remain flexible enough to conform to unexpected applications or overlaps?
    • A suggestion was made to speak about Product types in terms of the phases of production workflow that they work in. If a product is used in a specific phase of production, there are certain things it should be able to do.
    • Does the current list of enumerated Specific Features cover validation of the specific features that we think are important for an “ACES” Product?
    • Might there be a way to leverage the evidence submitted for product qualification to be used as documentation or per-product demonstrations? e.g. if screencasts demonstrating how features are implemented, could these be shared to help users?
1 Like

In last week’s meeting, we discussed things related to these prompts:

  • What is good, if anything, about the current qualification process?
  • What is bad, if anything, about the current qualification process?
  • What, if anything, should the ‘process’ be?
  • How can a Logo “Program” bring value to the manufacturer?

Here were some of the conversation topics we covered, in no particular order and roughly categorized. There is a lot of paraphrasing here, so if you want full context the meeting recording is available.

General

  • There should be a list of all the ACES components/features that a product might need/want to implement, with a few examples of common scenarios and what we might expect tools in certain categories to provide. This would be like an à la carte menu from which a product could decide which pieces are relevant to their product’s function and features. This “choose your own” product category approach recognizes that the manufacturer best knows their product’s function and features, and does not try conform to hard-preset categories.

  • Do we need ‘certification’ from the ACES staff?
    A logo could just be awarded after a product has brought its ACES implementation to market and it has been tested and confirmed by the user community.

  • The ACES user community has grown. Users reporting ‘I’ve successfully been able to do an ACES job with this product’ is valuable info for the community at large. This is trust building in products through the sharing of positive experiences when using ACES. The community is also adept at catching bugs and demanding that they be fixed.

  • The priorities and rationale for the ACES Logo Program might have shifted as the system has matured. The initial goal when ACES and the Logo Program first launched largely seemed to largely be for marketing. However, now that ACES has gained traction and matured, the Logo should be used to indicate that ACES components are available to use in a particular product and that it is a quality implementation.

Tests

  • Generally, if a vendor is going through the effort of implementing ACES features, they are most likely building and running some sort of tests. Self-applied tests work well.

  • The odds of applicants trying to ‘fake’ an ACES implementation are low. A poor implementation would become obvious to users and they would be held accountable on ACESCentral or elsewhere by users.

  • There are three important aspects:

    • the math
    • UI/UX consistency - e.g. familiarity to a user across products
    • interoperability - e.g. output from one tool feeds into another and works; ACES behaves the same regardless of the tool
  • Sometimes the tolerances for some of the requirements are super tight. The images look correct for normal values but some of the full range half float test images can give errors at the edges that can make it very hard to meet tolerances. Product teams then need to then weigh how much time and resources to dedicate to engineering such that the tolerances can be met and so test charts look perfect, even though normal range imagery looks correct.
    Is there or can there be room for exceptions to specific tolerances so long as the pictures look right? Are real world pictures more practical validations of implementations rather than test charts?

  • There are some existing tests/validiation available w/ ACES 1.0 and current Product Partner Logo Program

    • The Academy does supply before/after images for every transform in the ‘aces-dev’ repository. There is one pictorial ‘normal’ image and one ‘test chart’ style image.
    • There are also ‘pipeline’ test images that test a series of transforms at once – simulating common processing chains (e.g. [input.exr] ACES > ACEScc > CDL > ACES > LMT > RRT > ODT > [output.tiff])
    • CLF, for example, also provides its own test files and an Implementation Guide
  • Provide test images and/or end-to-end pipeline tests for manufacturers to use, if they desire, as well as being available for users, if they need to test or validate a product

  • Organize plug-fests

Current ‘Logo’d’ companies

  • The practice of listing companies under the Logo Program just for ‘signing up’ should be ended. It is super confusing why some of these companies are listed even though they do not have an ACES implementation in their product.

  • For the short term, we could break up the list to highlight the companies that have actually completed the Logo Qualification process for their product.

  • Longer term, companies should only get the logo if they already have a working ACES implementation.

OpenColorIO

  • How can OpenColorIO play a role to provide a practical and/or more accessible platform to test against?

  • Can or should OpenColorIO (v2.0+) be ‘blessed’ as a reference implementation?

I have taken a stab at a “strawman” for the Product Partner program. I started by cutting the existing Product Partner doc down to its bare minimum, and attempted to integrate some of the points that have been made in the past two meeting discussions. I have put it into a Google doc so that it can be a living document and so people can comment directly on items without us needing to pass around revisions of Word docs.

Note that the document is very preliminary and not in any way intended to be perceived as the final intention of this group. This doc aims solely to add structure to the conversations that have already occurred and to help guide and focus continued conversation and refinement of ideas previously presented.

There are many questions to be answered, including how much detail to include in requirements and, most importantly, defining the “how” such a program is going to run and be maintained long-term.

I mention in my comments that are already attached to the doc that I think we can leverage or cite TB-2014-002. I would recommend group participants peruse this document prior to Tuesday’s meeting to help inform discussion about defining User Experience requirements.

2 Likes