[DRAFT] glTF interactivity and flow graph update (For snapshot)

[RaananW]

I open this PR just to update the snapshot.
No reviews needed, as it will be closed right after.

[hybridherbst]

It seems I'm not yet allowed to upload files on the forum, so here's some more failing test files. They work in the Graph Authoring Tool implementation. Upon clicking the blue button the traffic light should go red-yellow-green-yellow-red.

• pointer/set with colors – results in wrong colors
  TrafficLight_Simple_SetColorPointer.glb.zip

• pointer/interpolate with colors – results in wrong colors
  TrafficLight_Simple_Interpolate.glb.zip

I believe these could have the same root issue.

[RaananW]

It seems I'm not yet allowed to upload files on the forum, so here's some more failing test files. They work in the Graph Authoring Tool implementation. Upon clicking the blue button the traffic light should go red-yellow-green-yellow-red.

• pointer/set with colors – results in wrong colors
  TrafficLight_Simple_SetColorPointer.glb.zip
• pointer/interpolate with colors – results in wrong colors
  TrafficLight_Simple_Interpolate.glb.zip

I believe these could have the same root issue.

Quick note on the files you provided, there is an error in the glTF that partly prevents us from parsing the types correctly (interpolation demo). The pointer's configuration is this:

"configuration":[{"id":"pointer","type":11,"value":"/materials/1/pbrMetallicRoughness/baseColorFactor"}], (expected float4)
and the value is this:

"values":[{"id":"value","type":7,"value":[0.9921569,0.7337824,0.0,1.0]}

value is a float4, but type:7 is float3. There are duplicated float3 as well, but we don't check that yet, so that's perfectly fine. So even if I make sure to parse vector4/vcolor4 correctly, this will fail. I will try finding a proper solution for this.

Anyhow, the set is already fixed (as I suspected it was an issue with vector-to-color), interpolation is being worked on.

[hybridherbst]

Thanks for fixing, good catch about float3/float4! – here's a file with the corrected type for color:
TrafficLight_Simple_SetColorPointer2.glb.zip

Not sure if the sandbox should already be updated – I can see the "red" color correctly but upon clicking the blue button the colors still go to some kind of dark purple.

I think the float3/float4 mixup might happen more often, as colors (typically float4 in all systems) can be either float3 or float4 in glTF, depending on usage, so I expect exporters to mess that up occassionally...

[RaananW]

Thanks for fixing, good catch about float3/float4! – here's a file with the corrected type for color: TrafficLight_Simple_SetColorPointer2.glb.zip

Not sure if the sandbox should already be updated – I can see the "red" color correctly but upon clicking the blue button the colors still go to some kind of dark purple.

I think the float3/float4 mixup might happen more often, as colors (typically float4 in all systems) can be either float3 or float4 in glTF, depending on usage, so I expect exporters to mess that up occassionally...

The sandbox will be updated in 15-20 minutes (waiting for the build to finish). Interpolation should also be working now.

The new file doesn't reset the colors to gray, so it only shows red yellow and green. Is that the correct behavior?

[hybridherbst]

Sorry, it was indeed not the correct file.

✅ This should be correct now (and works in the updated sandbox – thanks!):

• TrafficLight_Simple_SetColorPointer.glb.zip

❌ The pointer interpolation file does not behave correctly yet –

• File: TrafficLight_Simple_Interpolate.glb.zip

• Expected	Actual
  Screen.Recording.2024-11-18.at.17.26.40.mov	Screen.Recording.2024-11-18.at.17.27.42.mov

• Problems:

  • Interpolation flickers (whenever a pointer interpolation starts, the value seems to "snap" somewhere else for a frame)
  • my guess is that pointers never "stop" on their own after being done interpolating – thus the transition backwards (green-yellow-red) doesn't have any effect. I believe that independent of whether the transition is already done or not, newer pointer/interpolate calls should beat older ones as per the spec

[RaananW]

• Interpolation flickers (whenever a pointer interpolation starts, the value seems to "snap" somewhere else for a frame)
• my guess is that pointers never "stop" on their own after being done interpolating – thus the transition backwards (green-yellow-red) doesn't have any effect. I believe that independent of whether the transition is already done or not, newer pointer/interpolate calls should beat older ones as per the spec

That is true, I am not tracking currently running interpolation. Not yet, at least.

[bjsplat]

WebGL2 visualization test reporter:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15810/merge/testResults/webgl2playwright/index.html

[RaananW]

• Interpolation flickers (whenever a pointer interpolation starts, the value seems to "snap" somewhere else for a frame)
• my guess is that pointers never "stop" on their own after being done interpolating – thus the transition backwards (green-yellow-red) doesn't have any effect. I believe that independent of whether the transition is already done or not, newer pointer/interpolate calls should beat older ones as per the spec

Both were fixed in the last commit. The issue was not the interpolation running already. The issue was reusing already-existing blocks, which was prevented deliberately. I believe there was a discussion about that a long time ago (and this is why it was implemented this way originally), but I can't find it in the current state of the specs. So - blocks can now be reused and re-triggered.
The flickering was because of an issue we had in our bezier curve implementation, and that was fixed as well.

[hybridherbst]

Here's a test file that contains additional nodes not known by the engine. My understanding as per the spec discussions is that these should be ignored, as they could be defined in extensions that the engine simply doesn't support at the moment.

SimpleMath_SubGeneric_Logging.glb.zip

Looks like this – the goal here is to support logging in multiple engines that have different nodes for logging:

Currently, it causes an exception because the node is not known.

[RaananW]

Here's a test file that contains additional nodes not known by the engine. My understanding as per the spec discussions is that these should be ignored, as they could be defined in extensions that the engine simply doesn't support at the moment.

Should be fine now.

[hybridherbst]

Thanks!
Here's another test file, this time one that "just works" for a change :)

Counter (simple standalone)_Calculator.glb.zip

[hybridherbst]

Here's two more currently failing test assets (errors in the console):
SnailRace.zip

Should look like this:

[RaananW]

Here's two more

math/select was not attached. Should work now (i.e. once the build is done)

[bjsplat]

You have changed file(s) that make changes to the playground.
You can test the snapshot here:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/PLAYGROUND/refs/pull/15810/merge/

The snapshot playground with the CDN snapshot (only when available):

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/PLAYGROUND/refs/pull/15810/merge/?snapshot=refs/pull/15810/merge

Note that neither Babylon scenes nor textures are uploaded to the snapshot directory, so some playgrounds won't work correctly.