DEV: fix docs, add requirements.txt

README.md

@@ -2,13 +2,13 @@
 
 ## Overview
 
-This is a minimal PyTorch implementation of [DreamFusion](https://arxiv.org/abs/2209.14988) and its variant [Magic3D](https://arxiv.org/abs/2211.10440), where we utilize [NerfAcc](https://github.com/KAIR-BAIR/nerfacc) to handle the neural rendering part.
+This is a minimal PyTorch implementation of [DreamFusion](https://arxiv.org/abs/2209.14988) and its variant [Magic3D](https://arxiv.org/abs/2211.10440), where we utilize [NerfAcc](https://github.com/KAIR-BAIR/nerfacc) as the neural renderer and [Stable Diffusion](https://huggingface.co/spaces/stabilityai/stable-diffusion)/[DeepFloyd IF](https://github.com/deep-floyd/IF) as the guidance.
 
-![](https://github.com/chinhsuanwu/dreamfusionacc/assets/67839539/4e274e05-c9c8-464e-abfb-29c0049f2833)
+![](https://github.com/chinhsuanwu/dreamfusionacc/assets/67839539/3793259d-b4a1-4c9e-83e9-fc0416eadf4c)
 
 It takes ~30mins to train on a single 3090.
 
-⚠️ Please use [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion) or [threestudio](https://github.com/threestudio-project/threestudio) for higher quality 3D generation, this repo is not optimized but a light-weight re-implementation.
+⚠️ Please use [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion) or [threestudio](https://github.com/threestudio-project/threestudio) for higher quality 3D generation. This repo is not well-optimized, but it is a lightweight implementation that might be a good starting point if you are interested.
 
 
 ## Installation
@@ -37,12 +37,12 @@ To train
 ```bash
 python train.py --config config/peacock.yaml
 ```
-You can find all controllable settings in those *.yaml files.
+You can find all controllable settings in the yaml file.
 After the training is done, run
 ```bash
 python test.py --config config/peacock.yaml
 ```
-to get 360 visualizations.
+to render 360 visualizations.
 
 
 
@@ -66,4 +66,11 @@ to get 360 visualizations.
 
 ## Acknowledgments
 
-This implementation is heavily based on [NerfAcc](https://github.com/KAIR-BAIR/nerfacc) and [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion).
+This implementation is heavily based on [NerfAcc](https://github.com/KAIR-BAIR/nerfacc) and [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion).
+
+<a href="https://github.com/KAIR-BAIR/nerfacc">
+<picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://user-images.githubusercontent.com/3310961/199083722-881a2372-62c1-4255-8521-31a95a721851.png" />
+    <img alt="tyro logo" src="https://user-images.githubusercontent.com/3310961/199084143-0d63eb40-3f35-48d2-a9d5-78d1d60b7d66.png" width="250px" />
+</picture>
+</a>

config/head.yaml

@@ -0,0 +1,55 @@
+# general
+text: "a highly detailed stone bust of Theodoros Kolokotronis"
+log: "log/head"
+
+guidance: "stable-diffusion"  # choices: "if", "stable-diffusion"
+sd_version: "2.1"  # only valid when using stable-diffusion
+
+# training parameters
+device: "cuda"
+seed: 42
+lr: 1.0e-2
+weight_decay: 0.0
+max_steps: 10000
+log_interval: 1000
+eval_interval: 1000
+save_interval: 1000
+train_dataset_size: 100
+train_h: 256
+train_w: 256
+eval_dataset_size: 5
+eval_h: 256
+eval_w: 256
+eval_chunk_size: 65536
+
+# radiance field
+backbone: "ngp"  # currently only support "ngp"
+grid_resolution: 128
+grid_nlvl: 1
+grid_update_interval: 10
+grid_update_ema_decay: 0.6
+use_normal_net: False
+use_bkgd_net: True
+density_bias_scale: 10
+offset_scale: 0.5
+
+# scene parameters
+aabb: "-1.0,-1.0,-1.0,1.0,1.0,1.0"
+near_plane: 0.1
+far_plane: 1.0e+10
+
+# render parameters
+n_samples: 512  # 1024 in the paper setting, reduce if encounter OOM error
+cone_angle: 0.0
+alpha_thre: 0.0
+
+# regularization
+use_shading: True
+shading_sample_prob: "0.125,0.125,0.75"   # "albedo", "textureless", "lambertian"
+
+use_orient_loss: True
+lambda_orient: 1.0e-2
+use_opacity_loss: False
+lambda_opacity: 1.0e-3
+use_entropy_loss: True
+lambda_entropy: 1.0e-3

config/lion.yaml

@@ -0,0 +1,55 @@
+# general
+text: "a metal sculpture of a lion's head, highly detailed"
+log: "log/lion"
+
+guidance: "stable-diffusion"  # choices: "if", "stable-diffusion"
+sd_version: "2.1"  # only valid when using stable-diffusion
+
+# training parameters
+device: "cuda"
+seed: 42
+lr: 1.0e-2
+weight_decay: 0.0
+max_steps: 10000
+log_interval: 1000
+eval_interval: 1000
+save_interval: 1000
+train_dataset_size: 100
+train_h: 256
+train_w: 256
+eval_dataset_size: 5
+eval_h: 256
+eval_w: 256
+eval_chunk_size: 65536
+
+# radiance field
+backbone: "ngp"  # currently only support "ngp"
+grid_resolution: 128
+grid_nlvl: 1
+grid_update_interval: 10
+grid_update_ema_decay: 0.6
+use_normal_net: False
+use_bkgd_net: True
+density_bias_scale: 10
+offset_scale: 0.5
+
+# scene parameters
+aabb: "-1.0,-1.0,-1.0,1.0,1.0,1.0"
+near_plane: 0.1
+far_plane: 1.0e+10
+
+# render parameters
+n_samples: 512  # 1024 in the paper setting, reduce if encounter OOM error
+cone_angle: 0.0
+alpha_thre: 0.0
+
+# regularization
+use_shading: True
+shading_sample_prob: "0.125,0.125,0.75"   # "albedo", "textureless", "lambertian"
+
+use_orient_loss: True
+lambda_orient: 1.0e-2
+use_opacity_loss: False
+lambda_opacity: 1.0e-3
+use_entropy_loss: True
+lambda_entropy: 1.0e-3

config/peacock.yaml

@@ -34,7 +34,7 @@ density_bias_scale: 10
 offset_scale: 0.5
 
 # scene parameters
-aabb: "-2.0,-2.0,-2.0,2.0,2.0,2.0"
+aabb: "-1.0,-1.0,-1.0,1.0,1.0,1.0"
 near_plane: 0.1
 far_plane: 1.0e+10
 

requirements.txt

@@ -0,0 +1,21 @@
+accelerate
+attrdict
+diffusers
+einops
+huggingface-hub
+imageio
+ninja
+numpy
+PyYAML
+scipy
+tokenizers
+torch==1.13.0
+torchaudio==0.13.0
+torchvision==0.14.0
+tqdm
+transformers
+sentencepiece
+
+# for nerfacc
+git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
+nerfacc -f https://nerfacc-bucket.s3.us-west-2.amazonaws.com/whl/torch-1.13.0_cu117.html