Collect required data for local and farm render jobs to have a 'colorspace' value specified

[BigRoy]

Changelog Description

This makes it so that for all render products generated from Houdini renderers and their render instances the colorspace is set to that of the scene_linear OCIO role in the currently set OCIO config.

By doing so, the transcoding plug-in is capable of using this as the source colorspace when needing to transcode from a source colorspace to a destination colorspace.

Additional info

Without this it'd mean that the transcoding would basically get ignored because it would not know about the source colorspace of the output image. It would get ignored here.

The logic involved with this definitely shows we have some API refactoring to do in ayon_core to make this much easier to access and do. For now I've tried to add in comments in the code here what instance data is being set and why for the farm instances. But ayon_core.pipeline.farm.pyblish_functions can definitely use some cleanup/restructuring to simplify things.

Testing notes:

Global color management should be enabled in core! (However, it may be good to do also do at least a test run with one of the renderers with global color management disabled to ensure we're not breaking that.)

1. Local and farm render submissions should work from Houdini for the different renderers:
   • Karma ROP
   • Mantra ROP
   • V-Ray ROP
   • Arnold ROP
   • Redshift ROP
   • USDrender (with any of the above renderers)
2. The transcoding plug-in should kick in according to ayon+settings://core/publish/ExtractOIIOTranscode profiles. Note that if you have no matching profiles it would of course still not transcode.
3. The output image (reviewable) should have matching colors according to the transcoding profile's expectations. E.g. when using Houdini's display/view settings the colors should match as close as possible (1:1, but keeping in account any encoding color shifts).

[moonyuet]

I tested with mantra ROP with local and farm rendering. Looks good. But the variant name is bit strange and it is named similarly with the instance of ass product type?(isn't it supposed to be mantraROPMain for example?)
With this PR:

With the develop:

[MustafaJafar]

As far as I know, this PR's topic brings many topics into the table:

1. Extract OIIO Transcode settings/configuration. (Note that we don't need it for review product type as output colorspace is managed by the OpenGL ROP and validated by validate_review_colorspace.py)
2. Deduce the output color space of Render ROPs.
3. Support AOV filtering similar to Extract Review similar to the review aov filter in deadline settings ayon+settings://deadline/publish/ProcessSubmittedJobOnFarm/aov_filter
4. which viewer we are using to open our renders and reviewable. (I'm DJV and have no idea if there's some configuration I should do.)

---

anyways, Here's a snippet of my transcode plugin settings.

Here's Mantra local render result on my end

Here I've changed the extension in my transcode settings to exr

[MustafaJafar]

Here's an error that I faced by accident. I assume it's not related to this PR.
I had $F5 in my file path. which led to this

INFO: oiiotool ERROR: read : File does not exist: "//storage/work/ayon_projects/Robo/Assets/Character/roboangry/work/cfx/pyblish/renders/mantra_ropTest\mantra_ropTest.1001.beauty.exr"

also,

RuntimeError: Executing arguments was not successful: "(['E:\\Ynput\\ayon-third-party\\client\\ayon_third_party\\downloads\\oiio\\oiiotool.exe', '--info', '-v', '-i:infoformat=xml', '//storage/work/ayon_projects/Robo/Assets/Character/roboangry/work/cfx/pyblish/renders/mantra_ropTest\\mantra_ropTest.1001-1002#.beauty.exr'],)"Error:
oiiotool ERROR: read : File does not exist: "//storage/work/ayon_projects/Robo/Assets/Character/roboangry/work/cfx/pyblish/renders/mantra_ropTest\mantra_ropTest.1001.beauty.exr"

Full command line was:

> oiiotool.exe --info -v -i:infoformat=xml //storage/work/ayon_projects/Robo/Assets/Character/roboangry/work/cfx/pyblish/renders/mantra_ropTest\\mantra_ropTest.1001.beauty.exr

[BigRoy]

• Extract OIIO Transcode settings/configuration. (Note that we don't need it for review product type as output colorspace is managed by the OpenGL ROP and validated by validate_review_colorspace.py)

This I think you're improving now with: ynput/ayon-core#853

• Deduce the output color space of Render ROPs.

Yes, however - non-critical for now because it seems all default to using Houdini's scene_linear role by default. So unless user is overriding it explicitly per output aov or whatever it should behave to our needs. Which is 99% of the production scenarios + this PR already improves it over what it was too. So I'd say, separate backlog issue at best.

• Support AOV filtering similar to Extract Review similar to the review aov filter in deadline settings ayon+settings://deadline/publish/ProcessSubmittedJobOnFarm/aov_filter

As far as I'm aware - this is already done for farm instances, yet not for "local render instances" I assume? Separate issue and should basically be part of cleanup and unifying the farm and local render behavior I'd say.

• which viewer we are using to open our renders and reviewable. (I'm DJV and have no idea if there's some configuration I should do.)

Any viewer capable of viewing with OCIO colorspaces should suffice. Recent DJV 2 versions should work, MRV2, OpenRV but also compositing software like Nuke or Fusion. What's important is that when you're viewing it - is that you use the same OCIO config as Houdini uses. (And of course set the colorspace values accordingly in the viewer as needed).

So, viewing in a viewer a transcoded image (e.g. with display/view applied) should look 1:1 the same as in Houdini without additional color management (it's basically the sRGB view to it) and even a simple photo viewer should display it (if it's e.g. a JPEG)

The raw render output image from Houdini should look 1:1 the same as in Houdini when you view it in other software using the same OCIO config as Houdini uses, and with the same display & view transform set.

[BigRoy]

> I tested with mantra ROP with local and farm rendering. Looks good. But the variant name is bit strange and it is named similarly with the instance of ass product type?(isn't it supposed to be mantraROPMain for example?)

@moonyuet uh... that's so odd! I can't think of a reason the code changes here introduce that issue. Nor can I reproduce this?

---

@MustafaJafar has my comment allowed you maybe to test this better / more in-depth to confirm whether things work as they should?

[moonyuet]

@BigRoy I tested with the built addon instead of dev mode, it works normally. I used the setting below
I tested with display and view transcoding type

It has some colorspace difference between Houdini and Nuke (render in Houdini, and load in Nuke) but if I changed to the color-picking mode which is with Output-sRGB, the colorspace is matched(I hope I am not color blind)


Same with the colorspace transcoding type


But I think it is already correct as the color-picking mode tells it is rendered in Output-sRGB

[BigRoy]

Thanks for testing - was that a Houdini "local render" or "farm render".
Any chance you could share a Publisher UI JSON log export and/or a Deadline Render log?

[moonyuet]

Thanks for testing - was that a Houdini "local render" or "farm render". Any chance you could share a Publisher UI JSON log export and/or a Deadline Render log?

I tested in local render first(I mean the comment above). I will test the farm render soon.

[moonyuet]

I faced the exact same colorspace issue in farm rendering with the one in local rendering, which the color-picking mode means correct colorspace when loading in Nuke.
The deadline log below:
deadline_log_for_view_transform_transcoding_option.txt
deadline_log_for_colorspace_transcoding_option.txt
Publish UI report for view transform transcoding option in farm
publish-report-240823-22-42.json
Publish UI report for view transform transcoding option in local
publish-report-240823-22-43.json

[BigRoy]

Thanks @moonyuet

So in the first log it:

C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_third_party_1.0.1\ayon_third_party\downloads\oiio\oiiotool.exe --nosoftwareattrib --colorconfig C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_ocio_1.1.1\ayon_ocio\configs\OpenColorIOConfigs\aces_1.2\config.ocio -i:ch=R,G,B,A D:/sh_boss_proj/interior_lenny/lenny_restroom/asset/work/generic/pyblish/renders/mantra_ropMain\mantra_ropMain.1001.exr --ch R=R,G=G,B=B,A=A --subimage 0 --iscolorspace ACES - ACEScg --ociodisplay ACES DCDM -o C:\Users\Kayla\AppData\Local\Temp\op_transcoding_1u4pzh6t\mantra_ropMain.1001.exr

Using display view with display "ACES" and view "DCDM". That view seems like an odd choice maybe? :) Did you specify that in settings, or?

In the second log it:

C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_third_party_1.0.1\ayon_third_party\downloads\oiio\oiiotool.exe --nosoftwareattrib --colorconfig C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_ocio_1.1.1\ayon_ocio\configs\OpenColorIOConfigs\aces_1.2\config.ocio -i:ch=R,G,B,A D:/sh_boss_proj/interior_lenny/lenny_restroom/asset/work/generic/pyblish/renders/mantra_ropMain\mantra_ropMain.1001.exr --ch R=R,G=G,B=B,A=A --subimage 0 --colorconvert ACES - ACEScg Output - sRGB -o C:\Users\Kayla\AppData\Local\Temp\op_transcoding_debjxi2w\mantra_ropMain.1001.exr

Correctly assuming your image is in ACES - ACEScg and converts it to Output -sRGB which technically may look different than what you see in Houdini view because Display+View transform may be a different transformation from ACES - ACEScg?

---

@moonyuet any idea where the DCDM "view" comes from in your case? Because in the OCIO config the default view should not be DCDM but sRGB?

If that view were sRGB my HOPE is that then the color transcoding is a 1:1 match between:

1. The render without transcoding, with no color corrections, but with Display+View transform applied
2. The render with transcoding, with no color corrections or transforms whatsoever - the raw image
3. The houdini viewport display

[moonyuet]

Thanks @moonyuet

So in the first log it:

C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_third_party_1.0.1\ayon_third_party\downloads\oiio\oiiotool.exe --nosoftwareattrib --colorconfig C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_ocio_1.1.1\ayon_ocio\configs\OpenColorIOConfigs\aces_1.2\config.ocio -i:ch=R,G,B,A D:/sh_boss_proj/interior_lenny/lenny_restroom/asset/work/generic/pyblish/renders/mantra_ropMain\mantra_ropMain.1001.exr --ch R=R,G=G,B=B,A=A --subimage 0 --iscolorspace ACES - ACEScg --ociodisplay ACES DCDM -o C:\Users\Kayla\AppData\Local\Temp\op_transcoding_1u4pzh6t\mantra_ropMain.1001.exr

Using display view with display "ACES" and view "DCDM". That view seems like an odd choice maybe? :) Did you specify that in settings, or?

It is an odd choice, but just want to test to make sure the setting works as expected.

In the second log it:

C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_third_party_1.0.1\ayon_third_party\downloads\oiio\oiiotool.exe --nosoftwareattrib --colorconfig C:\Users\Kayla\AppData\Local\Ynput\AYON\addons\ayon_ocio_1.1.1\ayon_ocio\configs\OpenColorIOConfigs\aces_1.2\config.ocio -i:ch=R,G,B,A D:/sh_boss_proj/interior_lenny/lenny_restroom/asset/work/generic/pyblish/renders/mantra_ropMain\mantra_ropMain.1001.exr --ch R=R,G=G,B=B,A=A --subimage 0 --colorconvert ACES - ACEScg Output - sRGB -o C:\Users\Kayla\AppData\Local\Temp\op_transcoding_debjxi2w\mantra_ropMain.1001.exr

Correctly assuming your image is in ACES - ACEScg and converts it to Output -sRGB which technically may look different than what you see in Houdini view because Display+View transform may be a different transformation from ACES - ACEScg?

@moonyuet any idea where the DCDM "view" comes from in your case? Because in the OCIO config the default view should not be DCDM but sRGB?

It is coming from the setting. I set the display and view as the screenshot showed below.

It could be that we need to match both colorspace as well as view and transform?

[BigRoy]

It is coming from the setting. I set the display and view as the screenshot showed below.

Could you try with:

• Use Display&View
• Display: ACES
• View: sRGB

(It should actually also work with keeping Display+View empty, then it should take it from the OCIO default from Houdini; but let's try it explicitly first).

---

It could be that we need to match both colorspace as well as view and transform?

No.

• When set to "Use Display&View" it only uses the display and view.
• When set to "Use colorspace" it only uses the colorspace setting and disregards the display and view attributes there.

@MustafaJafar is working on improving the settings UI for that here: ynput/ayon-core#853

[moonyuet]

Could you try with:

• Use Display&View
• Display: ACES
• View: sRGB

I tried with the setting

when getting into nuke, the input transform is with default scene linear so the color should be lighter

But if we are setting at the same colorspace, it matches with the houdini render
So looks good.

• When set to "Use Display&View" it only uses the display and view.
• When set to "Use colorspace" it only uses the colorspace setting and disregards the display and view attributes there.

Without any display & view, with the color-picking input(Output-sRGB)

With the display & view filled and colorspace slot being empty in ayon setting, and the input transform is in color-picking mode.

[BigRoy]

So if all looks good based on this test by @moonyuet (thanks so much!) that means this may be approved and is ready for merge @MustafaJafar ?

[moonyuet]

So if all looks good based on this test by @moonyuet (thanks so much!) that means this may be approved and is ready for merge @MustafaJafar ?

it looks good for me(I just forgot the approve button)

[MustafaJafar]

I think I have a better idea now.
Here's a test from Houdini (Local Render).
My OIIO Transcode settings (I'm currently on my PR ynput/ayon-core#853)

DEBUG: Conversion command: E:\Ynput\ayon-third-party\client\ayon_third_party\downloads\oiio\oiiotool.exe --nosoftwareattrib --colorconfig E:\Ynput\ayon-ocio\client\ayon_ocio\configs\OpenColorIOConfigs\aces_1.2\config.ocio -i:ch=R,G,B,A //storage/work/ayon_projects/Robo/Assets/Character/roboangry/work/cfx/pyblish/renders/mantra_ropTest\mantra_ropTest.1001-1002#.beauty.exr --ch R=R,G=G,B=B,A=A --subimage 0 --colorconvert ACES - ACEScg Output - sRGB -o C:\Users\MUSTAF~1\AppData\Local\Temp\op_transcoding_zn78lunc\mantra_ropTest.1001-1002#.beauty.jpeg

Render View in Houdini	Loaded Render in Nuke	Loaded Review in Nuke	Loaded Review in DJV	Uploaded Review in AYON

I like how they are matched.

About the product naming:
I think it happens because I kept the same values I used for testing #67