Yuliang Xiu · Jinlong Yang · Dimitrios Tzionas · Michael J. Black

• [2022/05/16] BEV is supported as optional HPS by Yu Sun.
• [2022/05/15] Training code is released, please check Training Instruction.
• [2022/04/26] HybrIK (SMPL) is supported as optional HPS by Jiefeng Li.
• [2022/03/05] PIXIE (SMPL-X), PARE (SMPL), PyMAF (SMPL) are all supported as optional HPS.
• [2022/02/07] is ready to use.

• Given an RGB image, you could get:
  • image (png): segmentation, normal images (body + cloth), overlap result (rgb + normal)
  • mesh (obj): SMPL-(X) body, reconstructed clothed human
  • video (mp4): self-rotated clothed human

ICON's intermediate results
ICON's normal prediction + reconstructed mesh (w/o & w/ smooth)

• If you want to create a realistic and animatable 3D clothed avatar direclty from video / sequential images
  • fully-textured with per-vertex color
  • can be animated by SMPL pose parameters
  • natural pose-dependent clothing deformation

3D Clothed Avatar, created from 400+ images using ICON+SCANimate, animated by AIST++

• testing code and pretrained models (*self-implemented version)
  • ICON (w/ & w/o global encoder, w/ PyMAF/HybrIK/BEV/PIXIE/PARE as HPS)
  • PIFu* (RGB image + predicted normal map as input)
  • PaMIR* (RGB image + predicted normal map as input, w/ PyMAF/HybrIK/BEV/PIXIE/PARE as HPS)
• colab notebook
• dataset processing
  • RGB/Normal Rendering
  • Visibility computing
• training codes
  • ICON (w/ & w/o global encoder, w/ SMPL-X)
  • PIFu* (RGB image + predicted normal map as input)
  • PaMIR* (RGB image + predicted normal map as input, w/ voxelized SMPL)
• evaluation on CAPE (Hard, Easy)

Please follow the Installation Instruction to setup all the required packages, extra data, and models.

Please follow the Data Preprocess Instruction to generate the train/val/test dataset from raw scans (THuman2.0).

Please follow the Training Instruction to train your own model using THuman2.0.

cd ICON/apps

# PIFu* (*: re-implementation)
python infer.py -cfg ../configs/pifu.yaml -gpu 0 -in_dir ../examples -out_dir ../results

# PaMIR* (*: re-implementation)
python infer.py -cfg ../configs/pamir.yaml -gpu 0 -in_dir ../examples -out_dir ../results

# ICON w/ global filter (better visual details --> lower Normal Error))
python infer.py -cfg ../configs/icon-filter.yaml -gpu 0 -in_dir ../examples -out_dir ../results -hps_type {pixie/pymaf/pare/hybrik/bev}

# ICON w/o global filter (higher evaluation scores --> lower P2S/Chamfer Error))
python infer.py -cfg ../configs/icon-nofilter.yaml -gpu 0 -in_dir ../examples -out_dir ../results -hps_type {pixie/pymaf/pare/hybrik/bev}

Comparison with other state-of-the-art methods
Predicted normals on in-the-wild images with extreme poses

@inproceedings{xiu2022icon,
  title={{ICON}: {I}mplicit {C}lothed humans {O}btained from {N}ormals},
  author={Xiu, Yuliang and Yang, Jinlong and Tzionas, Dimitrios and Black, Michael J.},
  booktitle={IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
  month = jun,
  year={2022}
}

We thank Yao Feng, Soubhik Sanyal, Qianli Ma, Xu Chen, Hongwei Yi, Chun-Hao Paul Huang, and Weiyang Liu for their feedback and discussions, Tsvetelina Alexiadis for her help with the AMT perceptual study, Taylor McConnell for her voice over, Benjamin Pellkofer for webpage, and Yuanlu Xu's help in comparing with ARCH and ARCH++.

Special thanks to Vassilis Choutas for sharing the code of bvh-distance-queries

Here are some great resources we benefit from:

• MonoPortDataset for Data Processing
• PaMIR, PIFu, PIFuHD, and MonoPort for Benchmark
• SCANimate and AIST++ for Animation
• rembg for Human Segmentation
• smplx, PARE, PyMAF, PIXIE, BEV, and HybrIK for Human Pose & Shape Estimation
• CAPE and THuman for Dataset
• PyTorch3D for Differential Rendering

Some images used in the qualitative examples come from pinterest.com.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 (CLIPE Project).

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.

MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB was a part-time employee of Amazon during this project, his research was performed solely at, and funded solely by, the Max Planck Society.

For more questions, please contact [email protected]

For commercial licensing, please contact [email protected]