Skip to content

A minimal PyTorch implementation of text-to-3D models, e.g., DreamFusion and Magic3D, based on NerfAcc

License

Notifications You must be signed in to change notification settings

chinhsuanwu/dreamfusionacc

Folders and files

NameName
Last commit message
Last commit date

Latest commit

5293d65 · May 14, 2023

History

11 Commits
May 14, 2023
May 14, 2023
May 14, 2023
May 14, 2023
Apr 2, 2023
Apr 2, 2023
May 14, 2023
May 11, 2023
May 14, 2023
May 14, 2023
May 11, 2023

Repository files navigation

DreamFusionAcc

Overview

This is a minimal PyTorch implementation of DreamFusion and its variant Magic3D, where we utilize Instant-NGP as the neural renderer and Stable Diffusion/DeepFloyd IF as the guidance.

It takes ~30min to train with Stable Diffusion and ~40min with DeepFloyd IF on a single 3090.

⚠️ This repo has not yet been well-optimized, e.g., memory usage and lack of refinement stage. However, it is still a good reference for its easy-to-follow and lightweight implementation. Please use stable-dreamfusion or threestudio for better quality 3D generation and mesh export.

Installation

git clone https://github.com/chinhsuanwu/dreamfusionacc.git
cd dreamfusionacc
pip install -r requirements.txt
Dependencies (click to expand)

Dependencies

  • torch
  • tinycudann
  • nerfacc
  • numpy
  • imageio
  • einops
  • diffusers
  • trainsformers

You may install the pre-built wheels for NerfAcc. Please check out here. This repo is built upon torch 1.13.0 + cu117.

Howtos

To train

python train.py --config config/peacock.yaml

You can find all controllable settings in the yaml file. After the training is done, run

python test.py --config config/peacock.yaml

to render 360˚ visualizations.

Citation

@article{poole2022dreamfusion,
  author = {Poole, Ben and Jain, Ajay and Barron, Jonathan T. and Mildenhall, Ben},
  title = {DreamFusion: Text-to-3D using 2D Diffusion},
  journal = {arXiv},
  year = {2022},
}

@inproceedings{lin2023magic3d,
  title={Magic3D: High-Resolution Text-to-3D Content Creation},
  author={Lin, Chen-Hsuan and Gao, Jun and Tang, Luming and Takikawa, Towaki and Zeng, Xiaohui and Huang, Xun and Kreis, Karsten and Fidler, Sanja and Liu, Ming-Yu and Lin, Tsung-Yi},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2023}
}

Credits

This implementation is heavily based on NerfAcc and stable-dreamfusion. Kudos to the authors for their amazing work!

tyro logo

About

A minimal PyTorch implementation of text-to-3D models, e.g., DreamFusion and Magic3D, based on NerfAcc

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages