Support for glTF-Draco files

[colintle]

Issue: glTF-Draco Compression Example Not Working

Description:
The provided example in the examples directory does not work when attempting to load a glTF model with Draco compression. Specifically, when loading the Buggy/glTF-Draco/Buggy.gltf model, the application fails to render the scene as expected.

Steps to Reproduce:

1. Clone the glTF-Sample-Models repository.
2. Ensure the resource_dir is correctly set to the path of the cloned repository.
3. Run the provided CubeModel example with the line:

   self.scene = self.load_scene("Buggy/glTF-Draco/Buggy.gltf")

Questions

1. Are there additional steps or dependencies required to enable Draco decompression in this example?
2. Should I manually include a Draco decompression library or ensure compatibility with a specific version of moderngl_window or pygltflib?
3. Can you provide guidance on how to correctly handle Draco-compressed glTF models in this example?

[einarf]

The gltf2 loader is from 2018/19 before pygltflib was a thing. We should instead make a new loader using pygltflib.

If pygltflib has some decompression utilities we can piggyback on meanwhile is something to explore. That's all I can say right now. It might be something that can be temp fixed.

[colintle]

The gltf2 loader is from 2018/19 before pygltflib was a thing. We should instead make a new loader using pygltflib.

If pygltflib has some decompression utilities we can piggyback on meanwhile is something to explore. That's all I can say right now.

pygltflib currently does not have a decompression functionality. I am using DracoPy to decompress but I am not sure how I can render my model after decompress. Below is my reference code:

import io
import numpy as np
from PIL import Image
import moderngl_window
from moderngl_window.context.base import WindowConfig
from moderngl_window.scene.camera import KeyboardCamera
from glft2Parser import glft2Parser


class GLTFRenderer(WindowConfig):
    gl_version = (3, 3)
    window_size = (800, 600)
    title = "GLTF Renderer"
    resource_dir = "."

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.wnd.mouse_exclusivity = True
        self.camera = KeyboardCamera(
            self.wnd.keys,
            fov=75.0,
            aspect_ratio=self.wnd.aspect_ratio,
            near=0.1,
            far=1000.0,
        )
        self.camera.velocity = 10.0
        self.camera.mouse_sensitivity = 0.01

        # Initialize the GLTF parser
        self.parser = glft2Parser(flipTexture=True, Y_UP=True)

        # Load the GLTF file
        file_path = r"C:\Users\co986387\Documents\generate-synthetic-images\ucf\0147_858530.6583696686_-5537397.529199226_3036197.2162786936.glb"
        self.parser.load(file_path)

        # Process the first mesh
        parsed_data = self.parser.parseDracoData(0)
        if parsed_data:
            self.positions = np.array(parsed_data['vertices'], dtype=np.float32)
            self.indices = np.array(parsed_data['indices'], dtype=np.uint32)
            self.uvs = np.array(parsed_data['texCoords'], dtype=np.float32) if parsed_data['texCoords'] else None

            # Load the texture
            texture_index = parsed_data.get('imageIndex')
            if texture_index is not None:
                image_data = self.parser.parseImage(texture_index)
                image = Image.open(io.BytesIO(image_data))
                image = image.convert("RGBA")  # Ensure texture is in RGBA format
                texture_data = image.tobytes()
                texture_width, texture_height = image.size

                # Create OpenGL texture
                self.texture = self.ctx.texture((texture_width, texture_height), 4, texture_data)
                self.texture.use()

            # Create OpenGL buffers for positions, indices, and UVs
            self.vbo = self.ctx.buffer(self.positions.tobytes())
            self.ibo = self.ctx.buffer(self.indices.tobytes())
            self.uv_vbo = self.ctx.buffer(self.uvs.tobytes()) if self.uvs is not None else None

            # Create a Vertex Array Object (VAO)
            self.vao = self.ctx.vertex_array(
                self.load_program(vertex_shader="shaders/basic.vert", fragment_shader="shaders/basic.frag"),
                [
                    (self.vbo, "3f", "in_position"),
                    (self.uv_vbo, "2f", "in_uv"),
                ] if self.uv_vbo else [
                    (self.vbo, "3f", "in_position"),
                ],
                self.ibo
            )

    def on_render(self, time: float, frame_time: float):
        """Render the scene"""
        self.ctx.clear(0.1, 0.1, 0.1)
        self.vao.render()


if __name__ == "__main__":
    moderngl_window.run_window_config(GLTFRenderer)

[einarf]

What issues are you running into there? Stack trace? Black screen?
Do you have the shaders? I don't see you setting the projection or view matrix on the shader/program likely leading to a black screen or some random garble.
Do you have some sample file that is sharable?

[colintle]

What issues are you running into there? Stack trace? Black screen? Do you have the shaders? I don't see you setting the projection or view matrix on the shader/program. Do you have some sample file that is sharable?

Yeah, I am running into black screen. My zip folder contains relevant code, .vert, .frag, and the draco-compressed .glb file.
issue_with_draco.zip

[colintle]

@einarf any updates to this?

[einarf]

I don't know where glft2Parser comes but I suspect you are only missing setting the matrixes on your program

    def on_render(self, time: float, frame_time: float):
        """Render the scene"""
        self.ctx.clear(0.1, 0.1, 0.1)
        self.ctx.enable(moderngl.DEPTH_TEST | moderngl.CULL_FACE)
        self.program["projection"].write(self.camera.projection.matrix)
        self.program["view"].write(self.camera.projection.matrix)
        self.program["model"] = glm.translate(glm.vec3(0.0, 0.0, -10.0))
        self.vao.render(moderngl.POINTS)

Right now your vertices are multiplied by zero-matrices and they get eaten alive by the multiply by zero black hole :)

[colintle]

I don't know where glft2Parser comes but I suspect you are only missing setting the matrixes on your program

    def on_render(self, time: float, frame_time: float):
        """Render the scene"""
        self.ctx.clear(0.1, 0.1, 0.1)
        self.ctx.enable(moderngl.DEPTH_TEST | moderngl.CULL_FACE)
        self.program["projection"].write(self.camera.projection.matrix)
        self.program["view"].write(self.camera.projection.matrix)
        self.program["model"] = glm.translate(glm.vec3(0.0, 0.0, -10.0))
        self.vao.render(moderngl.POINTS)

Right now your vertices are multiplied by zero-matrices and they get eaten alive by the multiply by zero black hole :)

Is it possible if I could get in a call with you some time this week? I tried this out and got the following error:

File "c:\Users\co986387\Documents\generate-synthetic-images\load_glb.py", line 81, in on_render self.program["model"] = glm.translate(glm.vec3(0.0, 0.0, -10.0)) ~~~~~~~~~~~~^^^^^^^^^ File "C:\Users\co986387\AppData\Local\anaconda3\envs\preprocess\Lib\site-packages\moderngl\__init__.py", line 415, in __setitem__ self._members[key].value = value ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\co986387\AppData\Local\anaconda3\envs\preprocess\Lib\site-packages\_moderngl.py", line 76, in value data = struct.pack(self.fmt, *value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ struct.error: pack expected 16 items for packing (got 4)

[einarf]

That was a typo on my part

program["model"] = glm.translate(glm.mat4(), glm.vec3(0.0, 0.0, -10.0))

It was setting a vector instead of a matrix. It just moves the object a little further away from the camera.

[colintle]

That was a typo on my part

program["model"] = glm.translate(glm.mat4(), glm.vec3(0.0, 0.0, -10.0))

It was setting a vector instead of a matrix. It just moves the object a little further away from the camera.

Weird. It is still making a similar error

File "c:\Users\co986387\Documents\generate-synthetic-images\load_glb.py", line 81, in on_render self.program["model"] = glm.translate(glm.mat4(), glm.vec3(0.0, 0.0, -10.0)) ~~~~~~~~~~~~^^^^^^^^^ File "C:\Users\co986387\AppData\Local\anaconda3\envs\preprocess\Lib\site-packages\moderngl\__init__.py", line 415, in __setitem__ self._members[key].value = value ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\co986387\AppData\Local\anaconda3\envs\preprocess\Lib\site-packages\_moderngl.py", line 76, in value data = struct.pack(self.fmt, *value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ struct.error: pack expected 16 items for packing (got 4)

[einarf]

write() is probably needed then

program["model"].write(glm.translate(glm.mat4(), glm.vec3(0.0, 0.0, -10.0)))

[colintle]

write() is probably needed then

program["model"].write(glm.translate(glm.mat4(), glm.vec3(0.0, 0.0, -10.0)))

Still a black screen for me

[colintle]

What is weird is that when I do moderngl.POINTS and without
self.ctx.clear(0.1, 0.1, 0.1) self.ctx.enable(moderngl.DEPTH_TEST | moderngl.CULL_FACE) self.program["projection"].write(self.camera.projection.matrix) self.program["view"].write(self.camera.projection.matrix) self.program["model"].write(glm.translate(glm.mat4(), glm.vec3(0.0, 0.0, -10.0)))
, I see a very small white dot

[einarf]

if you can provide me this glft2Parser module I will be able to look at it locally

[colintle]

if you can provide me this glft2Parser module I will be able to look at it locally

Sounds good.. I attached a zip folder with all the relevant code. It would be easier to get in a call with you on Zoom, Discord, etc. What is your email so I can contact you outside of Github?

gltf2Parser.zip

[einarf]

here's a standalone version using the decoded mesh. Rendering works fine.
test.zip

[colintle]

here's a standalone version using the decoded mesh. Rendering works fine. test.zip

How did you get the decoded mesh?

[einarf]

Potential issues with your current code:

# wrong because we use the projection matrix as the view matrix
self.program["view"].write(self.camera.projection.matrix)
# This is correct
self.program["view"].write(self.camera.matrix)

Also you are missing the input event functions to control the camera.
Another potential issue might be that the index buffer could be 16 or 32 bit. Look at the data itself. You need to specify the index_element_size argument to the vertex array if they are not 4 byte integers.

[einarf]

I decoded it by manually reading bytes from the offsets mentioned in the json part of the glb file and passed those bytes into DracoPy.decode to obtain a DracoMesh. Then I just dumped the values as strings into some temp files and included them in the test.py file.

I suspect your current code might work if you fix the issues I pointed out. Compare with my version

[colintle]

I decoded it by manually reading bytes from the offsets mentioned in the json part of the glb file and passed those bytes into DracoPy.decode to obtain a DracoMesh. Then I just dumped the values as strings into some temp files and included them in the test.py file.

I suspect your current code might work if you fix the issues I pointed out

I am new to ModernGL and OpenGL in general so I will have a couple questions here and also in the future:

1. Did you use gltf2Parser to decode the glb file or you used your own code?
2. How would the difference between 16 and 32 bit affect the index buffer?

[einarf]

I decoded it by manually reading bytes from the offsets mentioned in the json part of the glb file and passed those bytes into DracoPy.decode to obtain a DracoMesh. Then I just dumped the values as strings into some temp files and included them in the test.py file.
I suspect your current code might work if you fix the issues I pointed out

I am new to ModernGL and OpenGL in general so I will have a couple questions here and also in the future:

1. Did you use gltf2Parser to decode the glb file or you used your own code?
2. How would the difference between 16 and 32 bit affect the index buffer?

1. Nope. I just opened the glb file and found the byte index to read from in the json part of it. It's probably smarter to use a parser.
2. Print out the index buffer and see if it displays sane values. It should be integers started from 0 to N were N is the number of vertex positions. Just print the numpy array and check its dtype. I assume dracopy will created it using the correct type (16 or 32 bit integer).

https://moderngl.readthedocs.io/en/5.8.2/reference/vertex_array.html

index_element_size=4 is default. If the index buffer is 16 bit you need to set it to 2 (bytes per element)

Also make sure you fix the other issues I pointed out. Possibly we can add optional support for draco in moderngl-window.

[colintle]

I decoded it by manually reading bytes from the offsets mentioned in the json part of the glb file and passed those bytes into DracoPy.decode to obtain a DracoMesh. Then I just dumped the values as strings into some temp files and included them in the test.py file.
I suspect your current code might work if you fix the issues I pointed out

I am new to ModernGL and OpenGL in general so I will have a couple questions here and also in the future:

1. Did you use gltf2Parser to decode the glb file or you used your own code?
2. How would the difference between 16 and 32 bit affect the index buffer?

1. Nope. I just opened the glb file and found the byte index to read from in the json part of it. It's probably smarter to use a parser.
2. Print out the index buffer and see if it displays sane values. It should be integers started from 0 to N were N is the number of vertex positions. Just print the numpy array and check its dtype. I assume dracopy will created it using the correct type (16 or 32 bit integer).

https://moderngl.readthedocs.io/en/5.8.2/reference/vertex_array.html

index_element_size=4 is default. If the index buffer is 16 bit you need to set it to 2 (bytes per element)

Also make sure you fix the other issues I pointed out. Possibly we can add optional support for draco in moderngl-window.

I will definitely try this out! Thank you for helping me out! I will keep in touch with you

[einarf]

I actually managed to load the glb file you shared with mglw's loader using the original example

[colintle]

I actually managed to load the glb file you shared with mglw's loader using the original example

I was able to address the issues you made and was also able to load it in:
load_glb.zip

I have a bunch of these "tiles", and am wondering if I can render all of them together in one window. Do you think that is possible?

[einarf]

That's more than possible. The quick way is just to load each individual one and render them with a different translation for the model matrix. If you have a few hundred of them that will work fine.

I suspect the translation value in these files are related to geoposition in some way

{
    "translation":[
        858524.87900817802,
        3036177.3827817831,
        5537361.978433162
    ]
}

The alternative is to merge the vertex data and index data into one big mesh. Should be easy enough if you remember to offset the index buffers.

I guess you also figured out that DracoPy returns points and uvs as 64bit floats

[colintle]

That's more than possible. The quick way is just to load each individual one and render them with a different translation for the model matrix. If you have a few hundred of them that will work fine.

I suspect the translation value in these files are related to geoposition in some way

{
    "translation":[
        858524.87900817802,
        3036177.3827817831,
        5537361.978433162
    ]
}

The alternative is to merge the vertex data and index data into one big mesh. Should be easy enough if you remember to offset the index buffers.

I guess you also figured out that DracoPy returns points and uvs as 64bit floats

Yeah the translation value are geoposition related: ECEF.. This is my first time doing this so I might ask questions along the way

[einarf]

I forgot : If you merge the meshes you will get in trouble when it comes to texturing and will need to use texture arrays instead. What's the scope here? How many are you rendering?

[colintle]

I forgot : If you merge the meshes you will get in trouble when it comes to texturing and will need to use texture arrays instead. What's the scope here? How many are you rendering?

Right now, around 300, but in the future, it could be up to thousands

[einarf]

You'll probably be fine for a while with the current method then. Just loop draw the "tile" and worry about mesh merging later IF performance ever turns into a problem. I guess you just need to figure out the right translation from those geopositions and make sure the camera is positioned in their view.

Maybe make a GeoTile type, throw them all in a list and iterate + tile.render() them.

For merging (if needed) you probably want a GeoChunk that contains 16 x 16 of these tiles and holds a single mesh and a texture array with 256 layers (a layer for each of the meshes). The chunk of course also needs a position.

It's not that complicated to pull off with a few pointers.

[colintle]

You'll probably be fine for a while with the current method then. Just loop draw the "tile" and worry about mesh merging later IF performance ever turns into a problem. I guess you just need to figure out the right translation from those geopositions and make sure the camera is positioned in their view.

Maybe make a GeoTile type, throw them all in a list and iterate + tile.render() them.

For merging (if needed) you probably want a GeoChunk that contains 16 x 16 of these tiles and holds a single mesh and a texture array with 256 layers (a layer for each of the meshes). The chunk of course also needs a position.

It's not that complicated to pull off with a few pointers.

I see I see.. I will try this out and let you know how it goes.. I am thinking of mesh merging since I am not sure how big the size of models I will be working on in the foreseeable future.. I have a couple questions:

1. If I were to merge all the meshes, would it be doable to place the camera based on a translation value?
2. Are there sample examples I can reference that does this mechanism of merging meshes?
3. How do the texture arrays come into play with merging the meshes?

[einarf]

Since the mesh is somewhat tilted this is actually a bit more complicted. I assumed it was axis aligned and started at 0, 0. I assume the geoposition when converted will align them well but you also need to adjust for the tilt? If this is part of a dataset there's probably some information somewhere about this.

It's probably better to expore merging later when you have more knowlege. There might also be other options.

[colintle]

Since the mesh is somewhat tilted this is actually a bit more complicted. I assumed it it was axis aligned and started at 0, 0. I assume the geoposition when converted will align them well but you also need to adjust for the tilt? If this is part of a dataset there's probably some information somewhere about this.

It's probably better to expore merging later when you more knowlege. There might also be other options.

Ah I see.. I will look into just doing the simpler approach then.. thanks!

[colintle]

Since the mesh is somewhat tilted this is actually a bit more complicted. I assumed it was axis aligned and started at 0, 0. I assume the geoposition when converted will align them well but you also need to adjust for the tilt? If this is part of a dataset there's probably some information somewhere about this.

It's probably better to expore merging later when you have more knowlege. There might also be other options.

@einarf I converted the positions to ECEF coordinates for the model so that I also use ECEF for the camera. I am not sure what else I need to see the model again
test.zip

[colintle]

Since the mesh is somewhat tilted this is actually a bit more complicted. I assumed it was axis aligned and started at 0, 0. I assume the geoposition when converted will align them well but you also need to adjust for the tilt? If this is part of a dataset there's probably some information somewhere about this.
It's probably better to expore merging later when you have more knowlege. There might also be other options.

@einarf I converted the positions to ECEF coordinates for the model so that I also use ECEF for the camera. I am not sure what else I need to see the model again test.zip

transformVertices.zip
So I was able to render the model with the updated/transformed verticles by using the model matrix. Now, when I get close to the model, it lags really bag. My GPU is NVIDIA RTX A5000 so I am not sure why it is lagging badly

[einarf]

A model like that shouldn't lag. What probably happens is that the translated position is too far out for a 32 bit floating point values causing very inaccurate transformations. The translation needs to be conveted into something that is compatible for realtime rendering.

moderngl-window master branch now supports draco-compressed meshes. If I load the file you provided I see the exact same problem. It runs at high fps but appears to lag because the translation value in the gltf file is just too high making the model rotation appear in large steps. Possibly using dmat types can help here but again, it's probably easie to just change the translated values?

[colintle]

A model like that shouldn't lag. What probably happens is that the translated position is too far out for a 32 bit floating point values causing very inaccurate transformations. The translation needs to be conveted into something that is compatible for realtime rendering.

moderngl-window master branch now supports draco-compressed meshes. If I load the file you provided I see the exact same problem. It runs at high fps but appears to lag because the translation value in the gltf file is just too high making the model rotation appear in large steps. Possibly using dmat types can help here but again, it's probably easie to just change the translated values?

Yeah so I just ignored that tbh. When you change the yaw and pitch through keyboard, it doesn't lag as much so I will stick to that

[einarf]

If you are displaying a group of patches in the same area I would pick one of them to be them to be the "origin" (0, 0) and subtract its position from the other patches. This way the matrices will work on acceptable floating point values. Something similar to this is a pretty common trick when rendering large worlds.

I don't think the translations in the gltf files was meant to be used as is.