On my other machine, I changed the link.txt file made by make to explicitly point to the libAcesContainer.dylib in /usr/local/lib. The build then completed (make, then make install), and I could run rawtoaces. When I did, and used -v -v -v verbosity, the terminal output indicated that the file had made its way through libraw and then there was no more output. The process completed, but there was no file written.
Something is amiss with aces_container. Is there any way to test the function of aces_container, independent of rawtoaces?
@kirkthibault you can test aces_container in conjunction with CTL. Try doing a brew install ctl then grab the ACES transforms from http://www.github.com/ampas/aces-dev. If you have an exr file on your machine and you want to convert it to an ACES 2065-4 file, run in through the unity CTL transform using the -format aces command on the output.
So the command will look something like this : ctlrender -ctl aces-dev/transforms/ctl/utilities/ACESutil.Unity.ctl inFile.exr outFile.exr -format aces
If that executes fine, do : exrheader outFile.exr
You should see a header that includes : acesImageContainerFlag (type int): 1
and
chromaticities (type chromaticities): red (0.7347 0.2653) green (0 1) blue (0.0001 -0.077) white (0.32168 0.33767)
That would verify that aces_container is installed and working properly.
Again, I am not too familiar with the need for files to access each other during compilation, but I wonder if this is somehow related to OS X Sierra sandboxing or some similar kind of silent problem.
OK! Progress. I tried manually editing the link.txt file again, replacing the following section at the end of the file:
-lAcesContainer
with the hardcoded path to the dylib:
/usr/local/lib/libAcesContainer.dylib
This is what I had done previously, and it did not work. However, I think that after linking to the homebrew version (as pictured in the screenshot above and suggested by Mio) and then performing this edit, compilation worked and an EXR was output.
For whatever reason, the shortcut reference to the dylib is not working, but replacing it with the literal path to the link works.
Yay for brute force!
Thanks for your help on this, I look forward to testing it.
kirk
FYI - the link.txt file to which I am referring is located in:
One initial observation I have made in the ten minutes I have tested - if you feed rawtoaces a raw file (for example a .CR2 canon raw file) it works and the EXR with “aces” added in the filename is created.
However, if you convert the raw file to a DNG and execute rawtoaces on the DNG, no EXR file is output.
Here is the terminal transcript for the conversion of a CR2 file:
Kirks-MacBook-Pro-2:EXR kirkt$ rawtoaces --mat-method 2 --wb-method 0 -v -v -v -v _MG_0112.CR2
Starting rawtoaces ...
Processing _MG_0112.CR2 ...
CB: Reading metadata pass 1 of 2 (data passed=Sample data passed)
Reading metadata finished
CB: Reading RAW data pass 0 of 2 (data passed=Sample data passed)
start_timevaling Reading RAW data (expecting 2 iterations)
CB: Reading RAW data pass 1 of 2 (data passed=Sample data passed)
Reading RAW data finished
The camera has been identified as a Canon EOS 5D Mark IV ...
White Balance method is 0 - Using white balance factors from file metadata ...
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw ...
CB: Scaling colors pass 0 of 2 (data passed=Sample data passed)
start_timevaling Scaling colors (expecting 2 iterations)
CB: Scaling colors pass 1 of 2 (data passed=Sample data passed)
Scaling colors finished
CB: Pre-interpolating pass 0 of 2 (data passed=Sample data passed)
start_timevaling Pre-interpolating (expecting 2 iterations)
CB: Pre-interpolating pass 1 of 2 (data passed=Sample data passed)
Pre-interpolating finished
CB: Interpolating pass 0 of 4495 (data passed=Sample data passed)
start_timevaling Interpolating (expecting 4495 iterations)
CB: Interpolating pass 506 of 4495 (data passed=Sample data passed)
CB: Interpolating pass 1012 of 4495 (data passed=Sample data passed)
CB: Interpolating pass 1518 of 4495 (data passed=Sample data passed)
CB: Interpolating pass 2024 of 4495 (data passed=Sample data passed)
CB: Interpolating pass 2530 of 4495 (data passed=Sample data passed)
CB: Interpolating pass 3036 of 4495 (data passed=Sample data passed)
CB: Interpolating pass 3542 of 4495 (data passed=Sample data passed)
CB: Interpolating pass 4048 of 4495 (data passed=Sample data passed)
CB: Converting to RGB pass 0 of 2 (data passed=Sample data passed)
start_timevaling Converting to RGB (expecting 2 iterations)
CB: Converting to RGB pass 1 of 2 (data passed=Sample data passed)
Converting to RGB finished
The IDT matrix is ...
0.622841, -0.048300, -0.104717
-0.445778, 1.231981, 0.309404
-0.068663, 0.262731, 0.773828
The final white balance coefficients are ...
1.950195 1.000000 1.444336
Writing ACES file to _MG_0112_aces.exr ...
Finished
and here is the transcript from the DNG conversion:
Kirks-MacBook-Pro-2:EXR kirkt$ rawtoaces --mat-method 2 --wb-method 0 -v -v -v -v _MG_0112.DNG
Starting rawtoaces ...
Processing _MG_0112.DNG ...
CB: Reading metadata pass 1 of 2 (data passed=Sample data passed)
Reading metadata finished
CB: Reading RAW data pass 0 of 2 (data passed=Sample data passed)
start_timevaling Reading RAW data (expecting 2 iterations)
CB: Reading RAW data pass 1 of 2 (data passed=Sample data passed)
Reading RAW data finished
The camera has been identified as a Canon EOS 5D Mark IV ...
White Balance method is 0 - Using white balance factors from file metadata ...
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw ...
CB: Scaling colors pass 0 of 2 (data passed=Sample data passed)
start_timevaling Scaling colors (expecting 2 iterations)
CB: Scaling colors pass 1 of 2 (data passed=Sample data passed)
Scaling colors finished
CB: Pre-interpolating pass 0 of 2 (data passed=Sample data passed)
start_timevaling Pre-interpolating (expecting 2 iterations)
CB: Pre-interpolating pass 1 of 2 (data passed=Sample data passed)
Pre-interpolating finished
CB: Interpolating pass 0 of 4483 (data passed=Sample data passed)
start_timevaling Interpolating (expecting 4483 iterations)
CB: Interpolating pass 506 of 4483 (data passed=Sample data passed)
CB: Interpolating pass 1012 of 4483 (data passed=Sample data passed)
CB: Interpolating pass 1518 of 4483 (data passed=Sample data passed)
CB: Interpolating pass 2024 of 4483 (data passed=Sample data passed)
CB: Interpolating pass 2530 of 4483 (data passed=Sample data passed)
CB: Interpolating pass 3036 of 4483 (data passed=Sample data passed)
CB: Interpolating pass 3542 of 4483 (data passed=Sample data passed)
CB: Interpolating pass 4048 of 4483 (data passed=Sample data passed)
CB: Converting to RGB pass 0 of 2 (data passed=Sample data passed)
start_timevaling Converting to RGB (expecting 2 iterations)
CB: Converting to RGB pass 1 of 2 (data passed=Sample data passed)
Converting to RGB finished
Finished
Thanks for the effort! Glad to hear that your method worked!
For “DNG” file, we have not implemented functions to covert DNG to ACES yet, so right now “rawtoaces” just passes DGN files. We will let you know when “rawtoaces” supports DGN.
Please let us know if you have any other questions.
It worked! Just to give you some details - I created a new clone of rawtoaces, replaced the CMakeLists.txt in root with your file from Dropbox. I created the build directory, entered it, did “cmake …” and then “make” and the make command was successful. BUT! Remember that previously I had linked aces_container to the Homebrew version per your recommendation in Post 17 above (in case that makes a difference to your solution).
Anyway, your solution worked for me - thank you!
As a related aside, OCIO has been implemented as a Photoshop plug-in - this is a great way of using rawtoaces to bring an EXR into photoshop as ACES and convert it to ACEScg (or whatever other form you want) for linear 32bit workflows in PS.
As a follow up - I cloned the most recent (as of 18 July 2017) rawtoaces git repository, with the new CMakeList.txt file and compilation was successful on my work machine, the original machine that was having problems.
Thank you for this fantastic tool! I just discovered it and have been playing around.
I successfully compiled it on Linux (Fedora 26) after tracking down all the dependencies. It works great for CR2 raw files.
I have a lot of DNG files as well from Magic Lantern dualiso processing. I was sad to realize that DNG files aren’t yet supported, and then excited to come across the above discussion.
I tried recompiling with the above CMakeLists.txt from a new clone of the git repo. It compiled successfully, but unfortunately here the behavior is still the same. The command appears to process but no file is output. Here is the output of the command being run.
$ rawtoaces -v -v -v -v --wb-method 0 --mat-method 2 _MG_0436-2.DNG
Starting rawtoaces …
Processing _MG_0436-2.DNG …
CB: Reading metadata pass 1 of 2 (data passed=Sample data passed)
Reading metadata finished
CB: Reading RAW data pass 0 of 2 (data passed=Sample data passed)
start_timevaling Reading RAW data (expecting 2 iterations)
CB: Reading RAW data pass 1 of 2 (data passed=Sample data passed)
Reading RAW data finished
The camera has been identified as a Canon EOS 60D …
White Balance method is 0 - Using white balance factors from file metadata …
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw …
CB: Scaling colors pass 0 of 2 (data passed=Sample data passed)
start_timevaling Scaling colors (expecting 2 iterations)
CB: Scaling colors pass 1 of 2 (data passed=Sample data passed)
Scaling colors finished
CB: Pre-interpolating pass 0 of 2 (data passed=Sample data passed)
start_timevaling Pre-interpolating (expecting 2 iterations)
CB: Pre-interpolating pass 1 of 2 (data passed=Sample data passed)
Pre-interpolating finished
CB: Interpolating pass 0 of 3458 (data passed=Sample data passed)
start_timevaling Interpolating (expecting 3458 iterations)
CB: Converting to RGB pass 0 of 2 (data passed=Sample data passed)
start_timevaling Converting to RGB (expecting 2 iterations)
CB: Converting to RGB pass 1 of 2 (data passed=Sample data passed)
Converting to RGB finished
Finished
I’m not sure if DNG support is still in the works, or if the above CMakeLists.txt no longer works due to recent changes in the code, but it would be great to have!
Thank you for your hard work on this project and let me know if I can help troubleshoot or test in any way!
You are right that we are not currently supporting DNG files. I was too busy to simply make the software stop further processing when the images are identified as “DNG” files
Please try the latest version of “rawtoaces” on the “feature/optimization” branch. We will merge it into the “master” soon. The processing will be halted right after the camera is identified. The speed is faster too.
I’ve included a normal dng from my Canon 60D digital slr camera, converted from Canon CR2 to DNG using standard settings on the Adobe DNG Converter software: canon_60d_lossless_compression.dng
And there’s a dng with lossy compression converted from a Sony A7s raw ARW file: sony_a7s_lossy_compression.dng
There’s also a couple of cinemadng files, one from the Black Magic Cinema Camera and one from the Axiom Beta (links in the sources.md file): blackmagic_cinema_camera_cinemadng.dng axiom_beta_video_camera.dng
Hopefully this is enough to start testing with. Thanks a lot for working on this and let me know if you need anything!
I am having some issues. The compile went well but when I am trying to convert a CR2 rawtoaces raises an assertionerror.
I already tried to unbind/uninstall/reinstall aces_container without any luck. I think aces container was already installed due to a ctl installation during the compile. Maybe that could cause the issue?
Here is the terminal transcript:
rawtoaces -H 1 --mat-method 2 -v -v -v -v /Users/JulianM/Desktop/acescg_sphere_test/_P0B9646.CR2
Starting rawtoaces ...
Processing /Users/JulianM/Desktop/acescg_sphere_test/_P0B9646.CR2 ...
The camera has been identified as a Canon EOS 5D Mark III ...
White Balance method is 0 - Using white balance factors from file metadata ...
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw ...
The IDT matrix is ...
0.649774, -0.075815, -0.116831
-0.434096, 1.256459, 0.250427
-0.066794, 0.228016, 0.690909
The final white balance coefficients are ...
1.000000 0.489250 0.745342
Writing ACES file to /Users/JulianM/Desktop/acescg_sphere_test/_P0B9646_aces.exr ...
Assertion failed: (outputBufferSize < 100e6), function configure, file /tmp/aces_container20151026-55213-ashfox/aces_container-1.0.1/aces_Writer.cpp, line 206.
Abort trap: 6
I downloaded rawtoaces and compiled it. It works well with the images I tried. I would like to request two things, though. I would like the ability to control the compression in the resulting exr file. Currently it exports a completely uncompressed file which can be huge compared to the raw original. Uncompressed exr is very rarely used in the wild since there are several lossless compression methods available in the exr specs. And sometimes using the dwaa lossy compression scheme can also be useful when images only serve as references.
Also, It would be nice to be able to choose AGEScg as the output colorspace for the images, since most VFX houses use that colorspace internally to store textures and plates.
I understand that these two changes break the compatibility with the SMPTE 2065-4 ACES container format. But I would certainly don’t mind if the acesImageContainerFlag was not set in those cases.
This is a tough one … I understand the desire to not have to deal with transcoding, but we’re really trying to make sure what gets interchanged is ACES 2065-1 images. Granted with EXR there’s metadata to help sort out issue, but we don’t want to end up back in the same boat where you’ve got to ask the sender “How is that file encoded?”
I think there’s food for though here on both sides. I’ve personally wondered if it might be advantageous if the EXR libraries allowed for transcoding on read/write. Idk … just an idea.
Please keep in mind that if the camera spectral sensitivity data is found to be available, you should still use the default setting to convert DNGs; otherwise, it is recommended to “–mat-method 1 --wb-method 0” for other DNGs.
Please test with more images and let us know how it works for you.