This is the official implementation of 2024 paper "What Does Stable Diffusion Know about the 3D Scene" by Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman, including the datasets about different physical properties introduced in the paper.
pip install pycocotools
pip install Pillow
pip install scipy
pip install -U scikit-learn
pip install ipdb
pip install scikit-image
Clone the github https://github.com/Tsingularity/dift/tree/main, and put the files under dift/ of this github. Use dift/dift_sd.py in this github to replace src/models/dift_sd.py. Then fill in the paths and
python dift/extract_dift_depth.py
For Same Plane and Perpendicular Plane: https://github.com/NVlabs/planercnn
For Material: https://github.com/apple/ml-dms-dataset
For Shadow: https://github.com/stevewongv/InstanceShadowDetection
For Occlusion: https://github.com/Championchess/A-Tri-Layer-Plugin-to-Improve-Occluded-Detection/tree/master and https://cocodataset.org/#home
For Support Relation and Depth: https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html
Row 1 for Occlusion and Row 2 for Depth
Train/Val/Test Image Names | Regions and Pairs
python SVM/depth_train_test_svm.py
Please cite our paper if you use the code/model/dataset of this github.
@article{zhan24physd,
author = "Guanqi Zhan and Chuanxia Zheng and Weidi Xie and Andrew Zisserman",
title = "What Does Stable Diffusion Know about the 3D Scene?",
booktitle = "arXiv",
year = "2024",
}