README.md

MatchNeRF

Official PyTorch implementation for MatchNeRF, a new generalizable NeRF approach that employs explicit correspondence matching as the geometry prior and can perform novel view synthesis on unseen scenarios with as few as two source views as input, without requiring any retraining and fine-tuning.

Explicit Correspondence Matching for Generalizable Neural Radiance Fields
Yuedong Chen¹, Haofei Xu², Qianyi Wu¹, Chuanxia Zheng³, Tat-Jen Cham⁴, Jianfei Cai^1
1Monash University, ²ETH Zurich, ³University of Oxford, ⁴Nanyang Technological University
arXiv 2023

Paper | Project Page | Code

Recent Updates

• 25-Apr-2023: released MatchNeRF codes and models.

---

Table of Contents

• Setup Environment
• Download Datasets
  • DTU (for both training and testing)
  • Blender (for testing only)
  • Real Forward Facing (for testing only)
• Testing
• Training
• Rendering Video
• Use Your Own Data
• Miscellaneous

Setup Environment

This project is developed and tested on a CUDA11 device. For other CUDA version, manually update the requirements.txt file to match the settings before preceding.

git clone --recursive https://github.com/donydchen/matchnerf.git
cd matchnerf
conda create --name matchnerf python=3.8
conda activate matchnerf
pip install -r requirements.txt

For rendering video output, it requires ffmpeg to be installed on the system, you can double check by running ffmpeg -version. If ffmpeg does not exist, consider installing it by running conda install ffmpeg.

Download Datasets

DTU (for both training and testing)

• Download the preprocessed DTU training data dtu_training.rar and Depth_raw.zip from original MVSNet repo.

• Extract 'Cameras/' and 'Rectified/' from the above downloaded 'dtu_training.rar', and extract 'Depths' from the 'Depth_raw.zip'. Link all three folders to data/DTU, which should then have the following structure

data/DTU/
    |__ Cameras/
    |__ Depths/
    |__ Rectified/

Blender (for testing only)

• Download nerf_synthetic.zip and extract to data/nerf_synthetic.

Real Forward Facing (for testing only)

• Download nerf_llff_data.zip and extract to data/nerf_llff_data.

Testing

MVSNeRF Setting (3 Nearest Views)

Download the pretrained model matchnerf_3v.pth and save to configs/pretrained_models/matchnerf_3v.pth, then run

python test.py --yaml=test --name=matchnerf_3v

If encounters CUDA out-of-memory, please reduce the ray sampling number, e.g., append --nerf.rand_rays_test==4096 to the command.

Performance should be exactly the same as below,

Dataset	PSNR	SSIM	LPIPS
DTU	26.91	0.934	0.159
Real Forward Facing	22.43	0.805	0.244
Blender	23.20	0.897	0.164

Training

Download the GMFlow pretrained weight (gmflow_sintel-0c07dcb3.pth) from the original GMFlow repo, and save it to configs/pretrained_models/gmflow_sintel-0c07dcb3.pth, then run

python train.py --yaml=train

Rendering Video

python test.py --yaml=test_video --name=matchnerf_3v_video

Results (without any per-scene fine-tuning) should be similar as below,

Visual Results

DTU: scan38_view24

Blender: materials_view36

Real Forward Facing: leaves_view13

Use Your Own Data

• Download the model (matchnerf_3v_ibr.pth) pretrained with IBRNet data (follow 'GPNR Setting 1'), and save it to configs/pretrained_models/matchnerf_3v_ibr.pth.
• Following the instructions detailed in the LLFF repo, use img2poses.py to recover camera poses.
• Update the colmap data loader at datasets/colmap.py accordingly.

We provide the following 3 input views demo for your reference.

# lower resolution but fast
python test.py --yaml=demo_own
# full version
python test.py --yaml=test_video_own

The generated video will look like,

Demo: own data, printer

Miscellaneous

Citation

If you use this project for your research, please cite our paper.

@article{chen2023matchnerf,
    title={Explicit Correspondence Matching for Generalizable Neural Radiance Fields},
    author={Chen, Yuedong and Xu, Haofei and Wu, Qianyi and Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
    journal={arXiv preprint arXiv:2304.12294},
    year={2023}
}

Pull Request

You are more than welcome to contribute to this project by sending a pull request.

Acknowledgments

This implementation borrowed many code snippets from GMFlow, MVSNeRF, BARF and GIRAFFE. Many thanks for all the above mentioned projects.