ClipID - linking ACES metadata to a clip

Following up on our discussion on yesterday’s call about ClipID:

Dailies and editorial systems keep track of links between clips and metadata (CDLs, LUTs, scene/take, etc) using their own databases, and then end up in EDLs, AAFs, XMLs in order to translate this info to online finishing tools.

This should still be possible with an AMF (ACES Metadata File).

However, it is possible that implementers may want to store information inside the AMF about the clip it is associated with.

<ClipID> is intended to be a place for that. It was discussed on the last call that this would be an optional element.

Example for an EXR image sequence (e.g. VFX pull):

<clipId>
        <clipName>A001C012</clipName>
        <sequence idx="#" min="1" max="240">A001_C012_AE0306_###.exr</sequence>
</clipId>

Please see the following schema:
https://github.com/aforsythe/amf/blob/51f03677267a1d5f4c21b510310257705d797ce3/acesMetadataFile.xsd#L118-L145

Is this sufficient?

I know @walter.arrighetti had some thoughts, and I seem to remember @ptr (Patrick from Pomfort) mentioning that timecode would also be useful for when a clip does not exist yet in a LiveGrade scenario.

Thanks,
Chris

Hi Chris,
Hopefully I can help a bit on this topic adding my two cents.

As we know, ClipID and TC are data that can be pulled from the SDI output of any major motion picture camera system. Therefore, if one wants to find a system on HOW TO LINK the AMF to the (will be) clips, we should definitely consider those two elements as anchor points, as well as other (optional) metadata like ClipUUID and or CRC32 hash that some camera systems have (which I think was part of @walter.arrighetti proposal)
There are, however, situations where either one can be compromised and/or not present. Let me try to cover some of them:

  • any camera that doesn’t have a SDI output, will not provide either of those data (i.e. GoPros, BlackMagic, DSLR cameras or any micro-camera of any sort). One can argue that those recording systems are not so important, but I do believe that in complex workflows with multiple cameras, the AMF needs to (especially) cover these kind of cameras as - considering its primary scope is to “define a color journey” - those cameras are the ones that always result more tricky in post production and are in higher need of a help when it comes to color pipelines;
  • any time you use a broadcast video transmitter (a part from SOME of the Teradek’s), SDI metadata (of any sort) get lost. Long range MPEG4 RF transmitters don’t even carry TC.
  • there are situations when a camera-generated clip requires to change name or TC, before proceeding into the defined pipeline: if the clip is shot at a different FPS from the project FPS, some approaches will suggest to re-wrap or re-encode the clip(s) at the project FPS, therefore changing the TC track. In every movie there are clips that change name: phones, DSLRs and some camera systems (Phantom, Flare), that don’t have unique naming conventions and therefore clips have to be renamed AFTER set, prior to the dailies.

It is clear that Aces Metadata Files will be mostly generated at the dailies stage, where some -or most- of those issues disappear, but if the scope is to find a way to use the AMF and to link it to clips from the very early on-set stages, then please do account for those scenarios as they might jeopardise a workflow based on the existence of those AMF files from set and therefore make everything unreliable.

I would like to suggest two (three) solutions:

  1. Definitely keep the ClipID and TC for sure, and possibly add the ClipUUID (like ARRI does) as another optional anchor point, BUT do not expect those data to come from set (especially the TC) and use them more as an informative data than an active data (they just suggest a link in a human readable way, software won’t use it necessarily)
  2. [THIS MIGHT BE MORE FOR THE IMPLEMENTATION GROUP, but…] Have a look at what RED does with the RMD files: the RED SDK suggests to the software reading the R3D file that if there is, within the same folder, a .RMD file with the same .R3D filename, then the software should read it and use it as metadata source. I would suggest we take the same approach: we give the AcesMetadataFile a “.amf” extension (which makes it even easier to find it within thousands of .xml that each project/software/computer has) and we name the .amf with the same video clip name. If, with each clip, there is a amf, then the file is used as a metadata source.
  3. [TO SOLVE THE ON-SET PROBLEM, which is not solved with point 2] Don’t expect a AMF to be linked to any clip at all at the on-set stage. Easy, no? Let me clarify: what we require on set from the AMF capabilities isn’t the automated clip-based color pipeline setup (the clip tells the software what to do), but a camera-based color pipeline automation (the software is educated to work on imageries in a specific way). In other words, we want to be sure that all Alexa cameras have a specific pipeline, all the RED cameras have another, all the Sony cameras have another and that we don’t mess up the incoming live feed by applying the wrong transform to it. So - and I mentioned this earlier- I suggest to introduce a sort of ACES Metadata Template (an “.amt”?) which would be, in essence, a Aces Metadata File without clip (a stray amf) which would be used to communicate a specific camera(source)-based color-pipeline to the on-set and dailies people where:
  • the xml structure is pretty much identical to the AMF, where there is an allocation for each stages of IDT-LMT-RRT-ODT, but there is a “locking mechanism” that only allows certain nodes of the pipeline to be modified by the user (for instance, the CDL);
  • The AMT can be used on livegrade applications (such as LiveGrade or Prelight) to pre-arrange a “neutral” look within the software (I had a brief talk about this with @ptr and I believe he was okey with the suggestion), so one can have - for instance- a Log-C_to_709.amt and a IPP2_to_709.amt that will automatically set up a look for an Alexa camera and a Red camera with - again, for instance- only the CDL adjustable. Once done that, the live grade software, if there is a presence of Clip ID or TC (or both) on the SDI feed, could generate AMF files with the right clipname. Or, if not, the AMT gets updated and re-saved as a new AMT look and manually passed over to the dailies lab or loaded into a software on-set that has access to the [now] generated clips and, like one would do with CDL’s now, link the new AMTs with one or multiple clips at the same time which will automatically generate AMF with the correct names, metadata and ClipID links;
  • The AMT is loaded directly inside the camera as a template and the camera system itself generates AMF for each clip, so the camera mag has already a bunch of AMF files correctly created as soon as the clip is recorded. And this mechanism could also allow for a live-integration like ARRI already does with the ARRI LOOK FILE 2, where the CDL can be updated even after loading the ARRI “.aml” in camera.

I’m not sure if this approach is of any interest, but I can assure you that it would make things easier on-set and would also allow us all to maintain a better control of the color pipelines across multiple units of the same project or between set and dailies/post labs. Also, we should never forget that interoperability across multiple (different) livegrading softwares and, even more, between live-grading software and dailies software, trying to set up a pre-determined (pre-agreed) color pipeline, is the key to solve 90% of the color-related issues we have now-days when it comes to transmit the intended look from set to post, to provide a visual feedback to the DP and Director or, even better, to view and expose the image through/from a monitor. The AMF can be the key to solve this issue.

Sorry for the long comment, hope it makes some sense.
Best regards,
FLG

1 Like

I definitely agree with Francesco.
Unfortunately, in general, we should never use filename alone – even when we know a file with filename exists.

In my extended AMF proposal (G-Drive link), from March 2019, I suggested three ways of linking (or non-linking) AMF with footage (slide #7) – alternatives for different real-world use-cases:

  1. Mainfest like: use of <clipID> anchoring AMF file using one or more file metadata among: filename, tapename/clipname/whatevername, TimeCode, even format-specific metadata (e.g. ARRIRAW or MXF UUIDs)
  2. Sidecar like: put one AMF file with the same name as the footage in the same folder (for on-set workflows that can only be linked to video streams rather than files – like Francesco pointed out). Further details in slide #6 as to whether the footage is a singl-file MOV or an EXR sequence.
  3. Enveloped like: one or more AMFs wrapped into the ISXD and/or the SCM components of an IMF package (useful for shot-by-shot AMFs).

Option 1 is particularly good for postproduction, where AMF may not be transported along with footage and different post applications may handle the color-pedigree very differently with each other.

Option 2 is particularly good for both on-set / near-set workflows and for watchfolder-based transcoders.

Option 3 is particularly good for inter-facility transfer and, of course, for archival / preservation.

Particularly as regards method #1 (using <clipID>), filename, tapename and timecode are useful, but using more metadata (possibly at least one per file-format) would be even more useful. Look at slide #8.

In my opinion it would be great if any camera manufacturer or file-format maintainer jumps in and indicates (at least) one metadata of their own file format to be uniquely referenced via <clipName> sub-element. I remember Joseph (ARRI) said he would look for ARRRIAW.
Otherwise the custom <metadata> sub-element should be used in a loosely specified way as a fallback.

Thank you both for your input on this topic.

I hope you can join our call tomorrow (Tuesday 8/20) at 9am.

I believe we will be doing a 90min call this time, in the hopes of nailing does the ClipID and framing questions.

Chris

Hi Chris,
Absolutely, I will be there!

Best
F

If I heard people on the call last week it sounded like there was a desire to move away from having a tag to associate the AMF file with specific content.

I believe @dtatut mentioned that much like ASC-CDL, AMF content associations might best be handled by other applications with robust mechanisms for keeping those associations.

Hi Alex.
Based on my experience in postproduction, the strongest reason why ACESclip (2014) was never implemented by any applications is that there was no usable and unambiguous way to link it with actual footage.

I strongly believe AMF can be also generated without (and before) any associated footage; that it is still useful without any reference to it – as pointed out in on-set usecases by @Giardiello and @ptr
Yet, current postproduction software will neither generate nor honor AMFs should <clipID> end up being, by any means, more ambiguous than those software’s propietary solutions: just look at how FilmLight extensively uses BLG.

Therefore I strongly believe that <clipID> should have stronger links than just a filename (that is really volatile), yet be an optional AMF element.

As a counter-example, DCP/IMF work so good because they aren’t just a bunch of files in one directory: there is a PKL file whose sole purpose is referencing, in one place, all the files of the package – using both their filenames and hashes (digests).
Oh, by the way: I have a couple of proposals to exploit AMF and IMF together.

As per @dtatut argument: ASC-CDL doesn’t take care of filename associations because it was defined in both XML, cmx-3600 and ALE “syntax”, therefore CDLs can be embedded in “any” .xml, .edl and .ale files. This doesn’t apply to AMF because only some products (not few – not many) can generate externally-referenced metadata in XML. Should that be the case, then you would just include the AMF’s aces: namespace into the XML and let some other pre-defined XML extensions take care of file associations. However, this is not in most of present usecases.

Looking forward to speaking with you all tomorrow.

Walter,

Thanks so much … great conversation that has to be had! I think your points make a lot of sense. I’m not sure if I was accurately capturing @dtatut point, but certainly the .edl and .ale methods of transporting CDL make it a slight different beast.

Looking forward to the discussion tomorrow

I’d like to recap what was my point of view on ClipID durng the last VWG, when the island connection went nuts and nobody could hear me any longer.

It seems there is an agreement about <clipID> element being optional. If there is none, the AMF is soft-linked to the footage that comes and goes in the same folder. Two use-cases for this:

  • On-set: DIT has a pre-determined AMF prepared where the camera magazine’s footage is going to be off-loaded.
  • Automated transcoders, where AMF can be prepared into a watchfolder together with any footage to be rendered into --say-- the AMF Output Transform’s colorimetry (or the AMF is just embedded into a larger XML from the transcoder’s own XSD).

The arguments for letting other files (ALE, EDL, etc.) do the binding between AMF and footage is good only for as long as such files use XML as well and an AMF’s <aces:colorPipeline> element can be fully embedded (along with its “namespace”) as an XML extension.
I’m thinking of IMF SCM and ISXD files, to Apple Final Cut fcpxml files, ARRI’s ARC XML workflows, and so on.

Otherwise, for non-XML files we need to have something like:

  • an AMF file (or more than one?)
  • footage/content file(s), and
  • one EDL/ALE file, binding the two

That looks a bit of too much old-style thing, isn’t it?.

If, howerver, there is a <clipID> element into the AMF, then the AMF is soft-linking one (or more) color pipeline(s) with a specific file (or file sequence, or file package).

In such a case, <clipID> is a bond with a filesystem object, therefore the soft-link shall depend on that file-format’s metadata.
Should the footage change either metadata, content and file-format, the <clipID> element in the AMF shall follow, by changing altogether, otherwise the binding is broken.

Therefore, when figuring out what goes into <clipID>, it’s important to have a list of possible options that depend on the file format. It’s logical and, in case an AMF-compatible application can’t playback or render a particular file format, then there’s no point in the application parsing how the AMF binds to that file-format via its specific metadata.
.
I hope this clarifies my position a bit better.

As I said a few times, in order to make AMF (ex ACESclip) really usable and actually used by products, it’s necessary that every file-format maintainers come up with candidate metadata to be used in <clipID> for their own format(s).
I call upon exr, ISO (mp4, png, jpg2k), SMPTE (mxf, IMF, dpx), Arri (ARRIRAW), RED (Redcode), Adobe (tiff, dng), Apple (QuickTIme), etc.

Now some proposal on how <clipID> can be practically populated, depending on the file format:

  • <file> for storing the filename of any single-file footage (MXF, QuickTime, MP4, etc.)
  • <seq> for storing the filename of a sequence of files. The # character is used, by default, for any single digits in the frame numbering, but can be optionally overridden with an optional idx argument. Optional min and max arguments may be used to bind an AMF to a particular “sub” frame range (that is useful if we have cuts within the same frame sequence, and multiple AMF per each cut); alternatively
  • <Id> for storing a DCP or IMF “clip” unique to a file package by referencing the package UUID…

Then, in addition to the above filesystem-specific elements, the following optional ones can be used:

  • <TimeCode> containing the clip’s starting TimeCode. If more than one TimeCode is supported a sring with the TC type is added in the optional type argument. Available TimeCode type names are filesystem-specific,
  • <clipName> containing the clip’s main tapename/clipname/reelname, as specified by file-format maintainer. If more of them are present within the same file format, then the rest can be specified in a sub-element
  • <modificationDateTime> the footage’s update time as reported in the filesystem. Applications should be lax anought to expected weigting in timezone differences, plus time drifts due to some filesystem’s time granularity.
  • a generic <metadata> element, with a mandatory attribute containing the name of one particular metadata specific to the given file format. If the fileformat’s metadata are explicitly named (e.g. in OpenEXR container), then there is no ambiguity at all.

Some example <clipID> practical usecases:

	<seq>movie_r3_hdgrd.master.#######.exr</seq>
	<metadata key="user.productionName">Movie Title</metadata>
	<metadata key="user.reelNumber">3</metadata>

In this case a whole sequence of EXRs is referenced by filename ony. Two named metadata are used to link further.

<Id>urn:uuid:f7b77066-2762-4483-a253-739524337266</Id>
		<metadata key="Annotation">MovieTitle_FTR-DVis_EN-XX_4K_20190309_VF</metadata>
		<metadata key="ContentTitle">MovieTitle_FTR-DVis_EN-XX_4K_20190309_VF</metadata>
		<metadata key="Creator">Clipster 6.0.0.7 (build 106320)</metadata>

In this case the binding of the whole AMF is with an IMF package’s own CPL, furthere referenced by the name of the IMP and, optionally, by the creator application’s name.

			<seq idx="#" min="1" max="3">A001_C012_AE0306_###.R3D</seq>
			<clipName>A001C012</clipName>
			<modificationDateTime>2014-06-20T12:24:03+02:30</modificationDateTime>

Here is a REDCODE example where the whole clip is recovered by means of a standard REDCODE metadata field, that the manufacturer should have pre-emptively declared as the standard/default for an AMF’s path filename indications.

There are three types of ClipID currently considered in the TLX timecode design [DRAFT]
for linking a timecode to a “SourceID"

24%20AM

a char string
a MAC address
a UUID

Thanks Jim,
In the ACES usecases:

  • Binding by MAC address is awesome for en-vogue IP-based streaming, and might have wide good usecases on-set, where AMF binds to camera-capture DIT equipment – provided that, as soon as the footage undergoes the next processing step, that can be moved into <history> and replaced with a file-based binding scheme.
  • Binding by a char string is good – as long as one knows unambiguously what does the char string means. Using a filename might be tempting, yet too loose.
  • Binding by UUID is, in my opinion, the elective way to go.

So I’d like UUID the most, but it’s unfortunately good only for those file formats supporting a “real” UUID field in their header. In particular:

  • some camera-native files (e.g. ARRIRAW) do;
  • some camera-native files have some “sort-of UID”, yet not a true UUID;
  • both MXF and DPX 2.0 do;
  • most non-camera related file formats I can think of (OpenEXR, TIFF, PNG, PSD) don’t

Therefore, in my opinion, even with TLX, we are still in need of a 2-columns table mapping one or more metadata filed per each file format, to be the elective tag fitting into <clipID> – be it either a char string, a UUID, or an on-set equipment’s MAC address.

@walter.arrighetti, this is to put a clarification here, and to answer a question that came up in the most recent AMF implementation meeting. There are two ‘unique ID’ fields in an ARRIRAW header, one a SMPTE UMID, the other a nominal UUID…but the latter is not really unique to the frame; all frames recorded with that clip have the same UUID.

When ARRIRAW frames our put through our ARRI Metadata Bridge SDK to map to OpenEXR attributes, there is an opportunity to generate true frame-level unique IDs by (for example) appending the frame count from the master timecode. This is not now being done. I will raise the issue. My guess is that if I put out a new release of the AMB SDK tomorrow, it would be adapted by all the ARRI partner program members by IBC, especially if (say) Netflix pushed vendors for this…but that would not address all the people running old releases of ARRIRAW -> OpenEXR software products.

Hi Joseph.
Thanks for the reply, that’s wonderful news !
The SMPTE UMID is a perfect candidate for an ARRI ClipID hook in to AMF, exaclty because it is clip-unique, rather than frame-unique.

Therefore, for an AMF referencing a piece of shot footage from ARRI camera via a <clipId> associating that SMPTE UMID, the colorpipeline(s) in th AMF are strongly and uniquely tied to the whole clip.

I have one question and one remark for you:

  • Is there SMPTE UMID preserved in ARRI-generated or ARRI-converted MXFs and ProRes’s as well ? If so, that would help both non-ARRIRAW workflows and the offline-editorial part of ARRIRAW ones, since both handle (rushes, etc…) in mxf/mov format.
  • Yes, it would be extremely good if the SMPTE UMID is preserved in ARRIRAW-to-EXR conversions, as well – although otherwise is not a big issue: here is why AMF already has potential features to overcome this.

Example: an application converts ARI to EXR. Even if SMPTE UMID is not preserved in OpenEXR, as long as the converting application is AMF aware, a new AMF will be generated along with the EXR’s, referencing this new file-sequence filenames and a new unique clip metadata inside the OpenEXR header – no matter if it’s not a UMID any longer.
In order to ensure that the clip is the same, the <history> element of AMF kicks-in, recoding that the current version of AMF which now references the EXR sequence with its own UID, but had been referring to an original ARRIRAW sequence with the SMPTE UMID in the past “historical section” of the AMF.

The above ideal example satisfies, in my opinion, all constrain of uniquely identifying and matching an AMF with the piece of footage, and sticking with it, from camera recording up to long-term archival of the finished master.

All in all, the association of a piece of footage with AMF can be done by using one or more of the below “hooks”:

  • Using filenames alone – this is just not strong/consistent enough.
  • Using an internal clip metadata (like SMPTE UMID) is perfect, as it sticks with footage even if it changes position in the filesystem or it’s renamed.

Summarizing:

  1. For as long as conversions are managed by AMF-aware applications, a new change for filename and/or file format and/or metadata hooked to the clipId can be handled gracefully by generating and AMF referencing the new footage (by filename and/or metadata) and keeping the old footage linked via filename and/or metadata in the <history> section.

  2. If conversions, rendering, file-transfer processes , are not all guaranteed to honour AMF, it’s paramount that the clipId metadata in the AMF is transported and preserved along the files, so the AMF will always reference that specific clip Identifier even in case the clip is renamed / format-converted / re-rendered.

Using both methdologies at the same time guarantees that either one sticks with the footage throughout the entire imaging pipeline: from set to archival.

The method #2 may not work if clipId is only a reference to filename, because, assuming not everything is AMF-aware, the binding would break as soon as somthing “touches” the filename and/or filesystem position of the footage.