** NeurIPS 2024 SPOTLIGHT **

3D Gaussian Splatting as Markov Chain Monte Carlo

Shakiba Kheradmand, Daniel Rebain, Gopal Sharma, Weiwei Sun, Yang-Che Tseng, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, Kwang Moo Yi

---

BibTeX

@inproceedings{kheradmand20243d,
    title = {3D Gaussian Splatting as Markov Chain Monte Carlo},
    author = {Kheradmand, Shakiba and Rebain, Daniel and Sharma, Gopal and Sun, Weiwei and Tseng, Yang-Che and Isack, Hossam and Kar, Abhishek and Tagliasacchi, Andrea and Yi, Kwang Moo},
    booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
    year = {2024},
    note = {Spotlight Presentation},
   }

Updates

Dec. 5th, 2024

A new change has been pushed to diff-gaussian-rasterization. In order to pull it:

cd submodules/diff-gaussian-rasterization
git pull origin gs-mcmc
cd ../..
pip install submodules/diff-gaussian-rasterization

This change incorporates "Section B.2 Tighter Bounding of 2D Gaussians" from StopThePop paper. This bound allows to fit a tighter bound around Gaussians when opacity is less than 1.

How to Install

This project is built on top of the Original 3DGS code base and has been tested only on Ubuntu 20.04. If you encounter any issues, please refer to the Original 3DGS code base for installation instructions.

Installation Steps

1. Clone the Repository:

   git clone --recursive https://github.com/ubc-vision/3dgs-mcmc.git
   cd 3dgs-mcmc

2. Set Up the Conda Environment:

   conda create -y -n 3dgs-mcmc-env python=3.8
   conda activate 3dgs-mcmc-env

3. Install Dependencies:

   pip install plyfile tqdm torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
   conda install cudatoolkit-dev=11.7 -c conda-forge

4. Install Submodules:

   CUDA_HOME=PATH/TO/CONDA/envs/3dgs-mcmc-env/pkgs/cuda-toolkit/ pip install submodules/diff-gaussian-rasterization submodules/simple-knn/

Common Issues:

1. Access Error During Cloning: If you encounter an access error when cloning the repository, ensure you have your SSH key set up correctly. Alternatively, you can clone using HTTPS.
2. Running diff-gaussian-rasterization Fails: You may need to change the compiler options in the setup.py file to run both the original and this code. Update the setup.py with the following extra_compile_args:

   extra_compile_args={"nvcc": ["-Xcompiler", "-fno-gnu-unique", "-I" + os.path.join(os.path.dirname(os.path.abspath(__file__)), "third_party/glm/")]}

   Afterwards, you need to reinstall diff-gaussian-rasterization. This is mentioned in 3DGS-issue-#41.

By following these steps, you should be able to install the project and reproduce the results. If you encounter any issues, refer to the original 3DGS code base for further guidance.

How to run

Running code is similar to the Original 3DGS code base with the following differences:

• You need to specify the maximum number of Gaussians that will be used. This is performed using --cap_max argument. The results in the paper uses the final number of Gaussians reached by the original 3DGS run for each shape.
• You need to specify the scale regularizer coefficient. This is performed using --scale_reg argument. For all the experiments in the paper, we use 0.01.
• You need to specify the opacity regularizer coefficient. This is performed using --opacity_reg argument. For Deep Blending dataset, we use 0.001. For all other experiments in the paper, we use 0.01.
• You need to specify the noise learning rate. This is performed using --noise_lr argument. For all the experiments in the paper, we use 5e5.
• You need to specify the initialization type. This is performed using --init_type argument. Options are random (to initialize randomly) or sfm (to initialize using a pointcloud).

How to Reproduce the Results in the Paper

python train.py --source_path PATH/TO/Shape --config configs/shape.json --eval