2DGS #208
2DGS #208
Conversation
There was a problem hiding this comment.
This is a wonderful start! I think it would be great to have similar numerical testing the same way that we have for the 3dgs, but we can coordinate this.
I'm inclined to separate the two into separate functions and interfaces, what are your thoughts @liruilong940607? There is also a lot of repeated code but that's not a big problem for now, we can figure out if/how we can share between 2dgs and 3dgs later once.
gsplat/cuda/csrc/rasterization.cu
Outdated
| // moving the channel padding from python to C side. | ||
| switch (channels) { | ||
| case 1: | ||
| rasterize_to_pixels_fwd_kernel<1><<<blocks, threads, 0, stream>>>( |
There was a problem hiding this comment.
rasterize_to_pixels_fwd_2dgs_kernel right?
There was a problem hiding this comment.
oops! Good catch!
gsplat/cuda_legacy/_wrapper.py
Outdated
| @@ -300,6 +293,10 @@ def rasterize_gaussians( | |||
| block_width: int, | |||
| background: Optional[Float[Tensor, "channels"]] = None, | |||
| return_alpha: Optional[bool] = False, | |||
| model_type: Literal["3dgs", "2dgs"] = "3dgs", | |||
There was a problem hiding this comment.
I think it's better to just make a separate function for 2dgs, but @liruilong940607 what do you think?
There was a problem hiding this comment.
yes that would be my preference as well. seems like 2D GS projection and 3D GS have different argument list, though some are shared, it would be a bit confusing to squeeze all arguments into the same function.
gsplat/cuda_legacy/_wrapper.py
Outdated
| @@ -548,6 +774,7 @@ def project_gaussians( | |||
| img_width: int, | |||
| block_width: int, | |||
| clip_thresh: float = 0.01, | |||
| model_type: Literal["3dgs", "2dgs"] = "3dgs" | |||
There was a problem hiding this comment.
here again, I feel like it'd be better to have separate functions
Yes totally agree. I definitely recommend start with developing as separate functions (or even separate files). This would make the development so much easier and faster without the need of considering compatibility and it won't interrupting what's already in there. I imagine some helper functions might be able to get reused, like on the python side, there is sorting related functions, and maybe also rasterization? The projection is definitely different between 2D and 3D GS so would be nice to have different interfaces. Merging is kinda like a cleanup, which we can certainly do it later. |
gsplat/cuda/csrc/projection.cu
Outdated
| glm::mat3 RS_camera = R * RS; | ||
| glm::mat3 WH = glm::mat3(RS_camera[0], RS_camera[1], mean_c); | ||
|
|
||
| glm::mat3 inverse_intrinsic = glm::mat3( |
There was a problem hiding this comment.
hmmm, the ellipse aabb in the original paper uses the projection matrix which transforms into clipping space, can we do this and directly take the same computation for ellipse aabb? @FantasticOven2 @liruilong940607 am I misunderstanding this?
|
I am highly interested in having this handy. |
|
Forward pass works on simple 2D experiments for v1.0 |
|
Slant views |
|
@FantasticOven2 I have tried to install latest version and it seems working after decreasing c.c. @liruilong940607 |
|
Oh I think the Could you remove that from this PR? Feels like maybe the |
Done! |
|
formatting as one last thing!! Thanks! |
Done! |
|
Format check is still failing. Could you run? Also, the |
the format is due to the black version mismatch; for the |
|
Thanks for the hard work! I want to ask can we replace the 3dgs to the 2dgs by simply replacing the btw, is there a bug or typo? gsplat/examples/simple_trainer_2dgs.py Line 454 in 2d6a967 It seems that no vars named renders in rasterize_splats().
Thanks! |
|
Thanks for the effort! Just a note, there is a missing option added in simple_trainer recently: "normalize_world_space" |
Hey, thanks for pointing this out; yes this is a typo. I will update this later on. Since inria implementation is mainly used for benchmark, it shouldn't affect the gsplat implementation. There are a few differences between |
|
Hello, Great work again. as I was running the code I have seen that when you set the absgrad to True it does not function and throw an error that there is no absgrad as a parameter for the densify. When I check the code it still only does it for means2d and I wonder if you plan to implement it also for the densift. Thanks. Have a nice day |
Hey! |
|
Hi @FantasticOven2, thanks for the great work! |
Hey, Thanks for your interest! The current 2DGS doesn't support camera pose gradient yet. We are working on additional features for 2DGS including camera pose gradient and will make an update in the next few month. |
|
Great work ! super interested in using for mesh extraction; any way to use the original "render.py" from the 2DGS repo from the pytorch checkpoint generated by this gsplat implementation? Or, plans to provide mesh extraction script here? Thanks! |
Hey! Thanks for your interest! Yes we do plan to add mesh extraction in the next one or two months. Meanwhile you can check Gaussian Opacity Field pull request as they have a draft mesh extraction |
|
great news! yep I have run that PR - works pretty well but need textures. looking forward to trying out your implementation. |
|
Hi @FantasticOven2, |
This is an attempt to integrate 2DGS https://surfsplatting.github.io/ into gsplat. You can run simple 2D traning example with:
2D verification examples:
python examples/2d_example.py --model-type {3dgs, 2dgs}Image fitting:
python examples/image_fitting.py --model-type {3dgs, 2dgs}3D scene reconstruction:
python examples/simple_trained_2dgs.py --data-dir ...traj_29999.1.1.mp4
Evaluations (Average over 7 scenes in Mipnerf-360 dataset)
w/o regularization
w/ normal consistency regularization
w/ normal consistency, distortion regularization