Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DRAFT] Gaussian Opacity Fields implementation #285

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

[DRAFT] Gaussian Opacity Fields implementation #285

wants to merge 17 commits into from

Conversation

niujinshuchong
Copy link

@niujinshuchong niujinshuchong commented Jul 16, 2024 •
edited
Loading

Implementation of Gaussian Opacity Fields. Training and testing works.

Benchmark results on Mip-NeRF 360 dataset.

PSNR SSIM LPIPS num_GS (M)
classic 28.989 0.871 0.136 3.23
GOF 28.843 0.867 0.140 3.08
GOF + depth-normal consitency 28.616 0.865 0.142 3.08

Visual comparison:
without depth-normal consistency
Screenshot from 2024-07-16 16-25-49
and with depth-normal consistency
Screenshot from 2024-07-16 16-25-56
Left: rendered normal. right: normal from depth.

Extracted Mesh:
ScreenCapture_2024-07-23-15-15-28

ScreenCapture_2024-07-23-15-15-54

TODO:

  • Mesh extraction
  • Clean up
  • Documation

@KuoFengYuan
Copy link

Hi,

I am trying to extract mesh in the gsplat module and have written some functions for this purpose, but I am encountering some bugs. Could you please guide me on how to better implement this feature? Any suggestions or example code would be greatly appreciated.

Thank you!

Bugs:
TSDF open3d function:
image

@niujinshuchong
Copy link
Author

niujinshuchong commented Jul 25, 2024 •
edited
Loading

@GuoFengYung You can refer to https://github.com/autonomousvision/gaussian-opacity-fields/blob/main/extract_mesh_tsdf.py for mesh extraction with TSDF. However, TSDF is not good at extracting fine detailed geometry or extracting the mesh at background regions. I just implemented GOF's tetrahedra meshing in this PR. You can try the latest code.

@KuoFengYuan
Copy link

KuoFengYuan commented Jul 25, 2024 •
edited
Loading

@GuoFengYung You can refer to https://github.com/autonomousvision/gaussian-opacity-fields/blob/main/extract_mesh_tsdf.py for mesh extraction with TSDF. However, TSDF is not good at extracting fine detailed geometry or extracting the mesh at background regions. I just implemented GOF's tetrahedra meshing in this PR. You can try the latest code.

Thanks for your reply!!
I will try it

@KohsukeIde
Copy link

Appreciate your amazing work!

I'm curious about how to proceed with from tetranerf.utils.extension import cpp. Would I need to clone and install the Tetranerf myself, or is there like a more convenient approach?

@niujinshuchong
Copy link
Author

@KohsukeIde You can install it from https://github.com/autonomousvision/gaussian-opacity-fields/tree/main/submodules/tetra-triangulation

@FantasticOven2 FantasticOven2 mentioned this pull request Oct 2, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@niujinshuchong @KuoFengYuan @KohsukeIde