This is the official code release of the CVPR 2024 paper NeuRAD: Neural Rendering for Autonomous Driving, building on top of Nerfstudio. Besides releasing the code for our NeuRAD model, we hope that this can lay the ground-work for research on applying neural rendering methods in autonomous driving.
In line with Nerfstudio's mission, this is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions.
Do you have feature requests or want to add your new AD-NeRF model? Or maybe provide structures for a new dataset? We welcome contributions!
The quickstart will help you get started with the NeuRAD model on a PandaSet sequence. For more complex changes (e.g., running with your own data/setting up a new NeRF graph), please refer to our references.
Our installation steps largely follow Nerfstudio, with some added dataset-specific dependencies. You must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.8 of CUDA. You can find more information about installing CUDA here.
NeuRAD requires python >= 3.10. We recommend using conda to manage dependencies. Make sure to install Conda before proceeding.
conda create --name neurad -y python=3.10
conda activate neurad
pip install --upgrade pipInstall PyTorch with CUDA (this repo has been tested with CUDA 11.7 and CUDA 11.8) and tiny-cuda-nn.
cuda-toolkit is required for building tiny-cuda-nn.
For CUDA 11.8:
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
# Some need to upgrade dill prior to tiny-cuda-nn install
pip install dill --upgrade
pip install --upgrade pip "setuptools<70.0"
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torchFor support of Waymo-Open-Dataset v2 (requires python3.10, also dependencies from this package are very strict so cannot add it to pyproject.toml and need install first):
pip install waymo-open-dataset-tf-2-11-0==1.6.1We refer to Nerfstudio for more installation support.
git clone https://github.com/georghess/neurad-studio.git
cd neurad-studio
pip install -e .OR if you want to skip all installation steps and directly start using NeuRAD, use the provided docker image or apptainer recipe:
Dockerfile or Apptainer recipe.
The following will train a NeuRAD model, our recommended model for real world AD scenes.
Begin by downloading PandaSet and unzip it under data/pandaset.
The dataset is no longer hosted by Scale but can be downloaded from the provided huggingface link.
Training models is done the same way as in nerfstudio, i.e.,
# Train model
python nerfstudio/scripts/train.py neurad pandaset-dataIf everything works, you should see training progress like the following:
Navigating to the link at the end of the terminal will load the webviewer. If you are running on a remote machine, you will need to port forward the websocket port (defaults to 7007).
If you run into issues, it could be due to the training taking up too much memory. You can try to adjust the model parameters according to the neurad-tiny vscode launch config.
It is possible to load a pretrained model by running
pyhton nerfstudio/scripts/train.py neurad pandaset-data --load-dir {outputs/.../nerfstudio_models}Given a pretrained model checkpoint, you can start the viewer by running
python nerfstudio/scripts/viewer/run_viewer.py --load-config {outputs/.../config.yml}Once you have a NeRF model you can render its output. There are multiple different renders, more info available using
python nerfstudio/scripts/render.py --helpBesides NeuRAD, we will provide a reimplementation of UniSim as well. Once this is released it can be trained using
# Train model
python nerfstudio/scripts/train.py unisim pandaset-dataFurther, as we build on top of nerfstudio, models such as nerfacto or splatfacto are available as well, see nerfstudio for details. However, note that these are made for static scenes.
For a full list of included models run python nerfstudio/scripts/train.py --help.
Each model contains many parameters that can be changed, too many to list here. Use the --help command to see the full list of configuration options.
python nerfstudio/scripts/train.py neurad --helpThere are four different methods to track training progress, using the viewer, tensorboard, Weights and Biases, and Comet. You can specify which visualizer to use by appending --vis {viewer, tensorboard, wandb, comet viewer+wandb, viewer+tensorboard, viewer+comet} to the training command. Simultaneously utilizing the viewer alongside wandb or tensorboard may cause stuttering issues during evaluation steps.
And that's it for getting started with the basics of NeuRAD. If you are missing some features, have a look at Planned Features to see if we have plans on implementing this. Otherwise, feel free to open an issue, or even better implement it yourself and open a PR!
If you want to add a dataset, look here. If you want to add a method, have a look here.
We have provided dataparsers for multiple autonomous driving dataset, see below for a complete list. However, your favorite AD dataset might still be missing.
To add a dataset, create nerfstudio/data/dataparsers/mydataset.py containing one dataparsers config class MyADDataParserConfig and one dataparser class MyADData. Preferrably, these inherit from ADDataParserConfig and ADDataParser, as these provide common functionality and streamline the expected format of AD data. For most datasets, it should then be sufficient to overwrite _get_cameras, _get_lidars, _read_lidars, _get_actor_trajectories, and _generate_dataparser_outputs.
| Data | Cameras | Lidars |
|---|---|---|
| 🚗 nuScenes | 6 cameras | 32-beam lidar |
| 🚗 ZOD (Annotations) | 1 camera | 128-beam + 2 x 16-beam lidars |
| 🚗 Argoverse 2 | 7 ring cameras + 2 stereo cameras | 2 x 32-beam lidars |
| 🚗 PandaSet (huggingface download) | 6 cameras | 64-beam lidar |
| 🚗 KITTIMOT (Timestamps) | 2 stereo cameras | 64-beam lidar |
| 🚗 Waymo v2 | 5 cameras | 64-beam lidar |
A brief introduction about Waymo dataparser for NeuRAD can be found in waymo_dataparser.md
Nerfstudio has made it easy to add new methods, see here for details. We have added our UniSim reimplementation as a plugin, which can be run as any other method using the ns-train command:
ns-train unisim pandaset-data --data data/pandasetand follow the instructions in the terminal.
See our UniSim repo for reference on how to add a new method as a plugin.
- Dataparser for multiple autonomous driving datasets including
- Dataparsing of lidar data (3D+intensity+time)
- Dataparsing of annotations
- Datamanager for lidar+image data
- Rolling shutter handling for ray generation
- Viewer improvements
- Lidar rendering
- Dynamic actor modifications
- NeuRAD - SOTA neural rendering method for dynamic AD scenes
- 3DGS implementation supporting dynamic objects
- UniSim plug-in
- Release code
- Collaboration friendly studio for NeRFs
You can find our paper on arXiv.
If you use this code or find our paper useful, please consider citing:
@article{neurad,
title={NeuRAD: Neural Rendering for Autonomous Driving},
author={Tonderski, Adam and Lindstr{\"o}m, Carl and Hess, Georg and Ljungbergh, William and Svensson, Lennart and Petersson, Christoffer},
journal={arXiv preprint arXiv:2311.15260},
year={2023}
}
