Strange culling behaviour when loading a custom glTF

[AliyaanZBH]

I am trying to render the Amazon Lumberyard Bistro scene that you can find on NVIDIA's ORCA site. The model comes with .FBX files and .DDS textures, I have managed to successfully convert the .fbx to a glTF and got it loading in the engine! The problem is that there seems to be inconsistent culling behaviour, perhaps relating to the mesh sorter, that I can't seem to pinpoint. I have attached some video examples that I think best demonstrate the effect but I will also embed some images here.

This is the view of the "back" half of the scene:

Note how the camera is tilted pretty far downward, this is because if you tilt any more upwards, you begin to get this effect:

I seem to have isolated it largely to moving the camera UP, however in certain parts of the scene, for example nearer to the center, the behaviour seems become more unpredictable, now clipping as you move the camera down aswell as left and right:

Interestingly, when switching to the OrbitalCamera, the clipping is significantly reduced, though still noticeable if you look for it. Here is an aerial view:

Please find video examples at the following links:
https://drive.google.com/file/d/1_8xUij-sasc6UGt2FIaO7md9AIWa-AjJ/view?usp=drive_link
https://drive.google.com/file/d/1rTDHt__CBC0KUoR2aTxRIe1bEmOqYFjx/view?usp=drive_link

I have tried messing around with different scales by resizing the model, as well as editing the near and far planes on the camera, but no luck so far. Any guidance here would be greatly appreciated

Related, I considered instead using the legacy renderer and loading the glTF instead of a .h3d there. I gave it an attempt, however certain textures don't apply correctly (I've narrowed atleast one issue down to not rendering cutout and transparency properly, such as for foliage and glass, but I'm unsure how to separate these using the ModelInstance class that glTFs get loaded as. Any advice here would also be appreciated!) and the model is rotated incorrectly. However, there is no clipping there! I wonder if that is a clue as to what is going on in the regular glTF renderer too!

One last helpful bit of info is that initially the scene loaded in with really busted glossiness values. It would appear pure chrome metallic unless the glossiness reduction value was set to 7 or higher, but this would result in the skybox turning black. I've since resolved that issue by removing the IBL bias from the globals struct when RenderMeshes is called, aswell as implementing the shadow map and frustum fix identified in this PR.

[stanard]

I think there is a bug with constructing the frustum from the projection matrix for culling. I just pushed a fix for orthographic camera frustum creation. I think this is a perspective camera though, so it shouldn't help. How can I reproduce this behavior? And have you tried to disable frustum culling to see if that's the issue?

[AliyaanZBH]

Strangely that frustum fix in the linked PR did help the material, here is what it looked like before:

To reproduce, first grab the Bistro glTF I created here. Then comment out the legacy renderer define, place Bistro folder next to ModelViewer and load through command line. I've only been testing the exterior so far, so I pass
-model Bistro/BistroExterior/BistroExterior.gltf

I then copy the method to construct the eye that is used for sponza2.gltf and enable the FlyingFPSCamera instead of the OrbitCamera. I have messed around with the camera Z range but that hasn't made any difference, I've tried it at the default 1-to-10000, I've tried 0.1-to-90000 etc.

I have just tried to disable frustum culling and it did do the trick! I'm unsure if I did this properly, I just removed the intersectSphere call inside Model::Render(). Thanks for that recommendation, but ideally frustum culling would work as it is a rather large scene and a performance improvement would be nice. Now that things aren't getting culled I've noticed a few other problems...

It seems like something weird is going on with the shadows:

Also, foliage appears with a black background when at a distance, implying perhaps something went wrong with alpha blend or test:

But then upclose, while it correctly handles the alpha, it seems to switch to the wrong LOD:

Any advice on these new issues?

[stanard]

Thanks. I will download your model and experiment with it myself. The shadow acne issue (as it's called) is probably due to front-face vs. back-face culling issues in the shadow map. Essentially, the surface is casting a shadow on itself, and the Z offset (if there is one) isn't large enough to prevent shadow bleeding.

[AliyaanZBH]

I managed to solve the shadow acne issue, turns out it was due to my earlier disabling of IBL bias when I was troubleshooting the chrome material issue. Re-enabling this removes the acne for me!

Interestingly, some of the foliage does render correctly. Maybe this indicates some kind of limit we've hit? It is an incredibly large scene.

The same applies for raillings in the scene, some work, some don't:

Also, I noticed that the scenes are much more CPU bound when using the glTF renderer versus the legacy one. Do you have any advice for optimisations I can make here?

[stanard]

I just pushed another fix that should correct the culling issue. The math for transforming bounding spheres was wrong. (Model.h and Model.cpp.) There are still lots of issues that I'd like to fix in this glTF renderer, but I don't have a lot of time right now. If you could get latest, you can see what's still broken and let me know here. I'll try to get to the issues when I have time. Thanks.

[AliyaanZBH]

That works great thank you! There's still the issue with alpha textures not blending out properly. Here is another example:

Any ideas on where to start?