Skip to content

Latest commit

 

History

History
151 lines (130 loc) · 5.63 KB

README.md

File metadata and controls

151 lines (130 loc) · 5.63 KB
Raw

TRACE's code for training

[Project Page] [Paper] [Video]

This is the code for training TRACE. For inference & evaluation, please refer to this instruction.

Installation

1.Preparing data.

Please download related project_data from here, unzip it and put it into ROMP/, such that ROMP/project_data/trace_data.

Prepare SMPL model files refer to this instrcution, and move all generated SMPL files from ~/.romp to ROMP/project_data/trace_data/parameters/. We are supposed to have

|-- parameters
|   |-- smpl
|   |   |-- SMPL_FEMALE.pkl
|   |   |-- smpl_kid_template.npy
|   |   |-- SMPL_MALE.pkl
|   |   |-- SMPL_NEUTRAL.pkl
|   |-- smil
|   |   |-- smil_web.pkl
|   |-- SMPLA_NEUTRAL.pth
|   |-- SMPL_NEUTRAL.pth
|   |-- smil_packed_info.pth

2.Please install some basic python libs, including

Pytorch with CUDA support
torchvision
Pytorch3D (please install it from source to match the installed pytorch version)
opencv-python
PIL Image

3.Except for the basic python libs, please install the dependences via

cd trace
sh install.sh

Datasets

Please prepare datasets for training using following links:
DynaCam: data
MPI-INF-3DHP: data (To merge the splited zip files, zip video_frames_split.zip -s=0 --out video_frames.zip)
Human3.6M: images and annotations (To merge the splited zip files, zip images.zip -s=0 --out h36m_images.zip)
3DPW: images annotations
PennAction: images annotations

Please follow the directory structure to organize them.

|-- datasets
|   |-- h36m
|   |   |-- images
|   |   |-- annots_smplkps.npz
|   |   |-- smpl_neutral_betas.npz
|   |   |-- h36m_train.txt
|   |   |-- h36m_test.txt
|   |-- mpi-inf-3dhp
|   |   |-- video_frames
|   |   |-- annots_video.npz
|   |-- DynaCam
|   |   |-- video_frames
|   |   |-- annotations
|   |-- 3DPW
|   |   |-- imageFiles
|   |   |-- annots.npz
|   |   |-- camera_annots.npz
|   |-- Penn_Action
|   |   |-- frames
|   |   |-- annots.npz

Finally, pleaset set the dataset root path:
If you put all datasets in one folder, then you just need to change dataset_rootdir in ROMP/trace/lib/config.py to the path of your dataset folder, like:

dataset_group.add_argument('--dataset_rootdir',type=str, default='/path/to/your/datasets/folder', help= 'root dir of all datasets')

If you put different dataset at different path, then you have to set them separately. For instance, to set the path of Human3.6M dataset, please change self.data_folder in ROMP/trace/lib/datasets/h36m.py to the path where you put Human3.6M, like

self.data_folder = /path/to/your/h36m/

Metadata

  1. Installing simple-romp:
pip install --upgrade setuptools numpy cython lap
pip install simple_romp==1.1.3
  1. Preparing SMPL model files in our format:

Firstly, please register and download:
a. Meta data from this link. Please unzip it, then we get a folder named "smpl_model_data" b. SMPL model file (SMPL_NEUTRAL.pkl) from "Download version 1.1.0 for Python 2.7 (female/male/neutral, 300 shape PCs)" in official website. Please unzip it and move the SMPL_NEUTRAL.pkl from extracted folder into the "smpl_model_data" folder.
c. Please also download SMIL model file (DOWNLOAD SMIL) from official website. Please unzip and put it into the "smpl_model_data" folder, so we have "smpl_model_data/smil/smil_web.pkl".
Then we can get a folder in structure like this:

|-- smpl_model_data
|   |-- SMPL_NEUTRAL.pkl
|   |-- J_regressor_extra.npy
|   |-- J_regressor_h36m.npy
|   |-- smpl_kid_template.npy
|   |-- smil
|   |-- |-- smil_web.pkl

Secondly, please convert the SMPL model files to our format via

# please provide the absolute path of the "smpl_model_data" folder to the source_dir 
romp.prepare_smpl -source_dir=/path/to/smpl_model_data
romp.prepare_smpl -source_dir=/path/to/smpl_model_data --gender=female
romp.prepare_smpl -source_dir=/path/to/smpl_model_data --gender=male

bev.prepare_smil -source_dir=/path/to/smpl_model_data
cp ~/.romp/smil_packed_info.pth ~/.romp/SMIL_NEUTRAL.pth

The converted file would be save to "~/.romp/" in defualt.

Please don't worry if there are bugs reporting during "Preparing SMPL model files". It is fine as long as we get things like this in "~/.romp/".

|-- .romp
|   |-- SMPL_NEUTRAL.pth
|   |-- SMPL_FEMALE.pth
|   |-- SMPL_MALE.pth
|   |-- SMPLA_NEUTRAL.pth
|   |-- SMPLA_FEMALE.pth
|   |-- SMPLA_MALE.pth
|   |-- smil_packed_info.pth
|   |-- SMIL_NEUTRAL.pth

Train

Please edit the settings in ROMP/trace/configs/trace.yml and then run

cd ROMP/trace
sh train.sh

Citation

@InProceedings{TRACE,
    author = {Sun, Yu and Bao, Qian and Liu, Wu and Mei, Tao and Black, Michael J.},
    title = {{TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments}}, 
    booktitle = {CVPR}, 
    year = {2023}}