What Does Stable Diffusion Know about the 3D Scene?

This is the official implementation of 2024 paper "What Does Stable Diffusion Know about the 3D Scene" by Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman, including the datasets about different physical properties introduced in the paper.

• paper link
• project page

Installation (Python 3.8.8 + Numpy 1.20.1 + PyTorch 1.13.1)

pip install pycocotools
pip install Pillow
pip install scipy
pip install -U scikit-learn
pip install ipdb
pip install scikit-image

Extract Stable Diffusion Feature

Clone the github https://github.com/Tsingularity/dift/tree/main, and put the files under dift/ of this github. Use dift/dift_sd.py in this github to replace src/models/dift_sd.py. Then fill in the paths and

python dift/extract_dift_depth.py

Download Original Datasets

For Same Plane and Perpendicular Plane: https://github.com/NVlabs/planercnn

For Material: https://github.com/apple/ml-dms-dataset

For Shadow: https://github.com/stevewongv/InstanceShadowDetection

For Occlusion: https://github.com/Championchess/A-Tri-Layer-Plugin-to-Improve-Occluded-Detection/tree/master and https://cocodataset.org/#home

For Support Relation and Depth: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

Download Our Datasets

Row 1 for Occlusion and Row 2 for Depth

Depth

Train/Val/Test Image Names | Regions and Pairs

Train and Test Linear SVM

Depth

python SVM/depth_train_test_svm.py

Experiment Results

Citation

Please cite our paper if you use the code/model/dataset of this github.

@article{zhan24physd,
author       = "Guanqi Zhan and Chuanxia Zheng and Weidi Xie and Andrew Zisserman",
title        = "What Does Stable Diffusion Know about the 3D Scene?",
booktitle    = "arXiv",
year         = "2024",
}