Implement Bilateral Guided Gaussian Splatting.

[jefequien]

Bilateral Guided Radiance Fields showed that bilateral grids are very effective at modeling exposure changes and other ISP effects. This PR ports their bilateral grid library onto Gaussian splatting. It seems quite effective at removing floaters in casually captured videos and improves color-corrected PSNR on the bilarf dataset.

compare_exp_opt.mp4

lionpavilion_compare.mp4

Original authors also expected this to work quite well. yuehaowang/bilarf#1

TODO:

• Match original paper's learning rate schedule exactly.

Does this belong in gsplat or is this more appropriate for nerfstudio?

[ichsan2895]

how to test it with gsplat-mcmc? Is it like this?

pip install git+https://github.com/jefequien/gsplat-mcmc.git@c4793ad154356d87065fa154aadbf37cf44e00db
python3 gsplat/examples/simple_trainer_mcmc.py

[jefequien]

Use simple_trainer_mcmc.py with the --exp_opt flag.

    python simple_trainer_mcmc.py --eval_steps $EVAL_STEPS --save_steps $SAVE_STEPS --disable_viewer --data_factor $DATA_FACTOR \
        --init_type sfm \
        --cap_max $CAP_MAX \
        --max_steps $MAX_STEPS \
        --data_dir $SCENE_FOLDER/$SCENE/ \
        --render_traj_path $RENDER_TRAJ_PATH \
        --exp_opt \
        --result_dir results/3dgs_exposure_opt/$SCENE/

[ichsan2895]

Great, I will test later. I will inform the result in renders channel in nerfstudio discord servers.

[ichsan2895]

I got error when run simple_trainer_mcmc.py --exp_opt --data_dir path/to/scene

Traceback (most recent call last):
  File "/workspace/GSPLAT_EXPOPT/gsplat-mcmc/examples/simple_trainer_mcmc.py", line 30, in <module>
    from gsplat.utils.lib_bilagrid import BilateralGrid, slice
ModuleNotFoundError: No module named 'gsplat.utils'

[jefequien]

Oops. Missed an __init__.py file. Added.

[ichsan2895]

Whats going on to this PR? It really nice feature implementation for gsplat :)

[jefequien]

@ichsan2895 The nerfstudio team is comparing it to appearance embeddings and working on adding it to splatfacto. @liruilong940607 will probably decide what to do with this PR in gsplat when he gets back from vacation.
nerfstudio-project/nerfstudio#3316

[henrypearce4D]

@ichsan2895 The nerfstudio team is comparing it to appearance embeddings and working on adding it to splatfacto. @liruilong940607 will probably decide what to do with this PR in gsplat when he gets back from vacation. nerfstudio-project/nerfstudio#3316

Super exciting, I'm eager to test processing a sequence of frames!

[SharkWipf]

Hmm. I realize this probably isn't the right place, but I thought the point of Gsplat was to implement all the base methods, and for Nerfstudio/Splatfacto to integrate the individual methods from Gsplat.
In which case things like this would be more at home in Gsplat I'd think.
Did I misunderstand the relation between Gsplat and Nerfstudio?

[jefequien]

Bilateral grids can be used to improve both nerfacto and splatfacto. Thus it makes sense for the method to live nerfstudio and share code between the two. This method exists entirely outside the core gsplat rasterization library but could live in gsplat for showcase/experimentation purposes.

[SharkWipf]

Ah, okay, makes sense, I didn't realize it applied to NeRFs too.
It's a bit odd how NeRFs are integrated and splats aren't, I guess it makes sense with how splats came later but it feels sub-optimal.
I guess in theory the most sensible design would be to split nerfs off as well, and having a gsplat library/project, a nerf library/project, a common shared library for things like this, and Nerfstudio to tie it all together as just a way to mix and match different methods/techniques. But that'd be a major refactor and probably would break all external methods.

[Tao-11-chen]

@jefequien Thanks very much for your great implementation. In my experiments, the grid sampling of large images will take unacceptable video memory because it will sample from a very large 5D grid with size (num_pixels_in_the_img, 12, 8, 16, 16), do you have any ideas on it? thanks in advance.

[Tao-11-chen]

@jefequien Thanks very much for your great implementation. In my experiments, the grid sampling of large images will take unacceptable video memory because it will sample from a very large 5D grid with size (num_pixels_in_the_img, 12, 8, 16, 16), do you have any ideas on it? thanks in advance.

The problem is solved with the custom grid_sample function, which can largely reduce memory consumption in the case of sampling from the same bilateral_grid(image).

[liruilong940607]

I think we could put this file under examples/, as this is not something falls into the umbrella of gsplat

[liruilong940607]

LGTM overall! Just a minor request on moving the lib_bilagrid.py to the examples/ folder

[liruilong940607]

Remove this util/ folder if we move the lib_bilagrid out

[jefequien]

Done

[liruilong940607]

LGTM! thanks Jeffrey for bring the bilateral grid into gsplat!