Avatar Image Generator

Faces Domain

Generated Cartoons

Based on the paper XGAN: https://arxiv.org/abs/1711.05139

The problem

This repo aims to contribute to the daunting problem of generating a cartoon given the picture of a face.

This is an image-to-image translation problem, which involves many classic computer vision tasks, like style transfer, super-resolution, colorization and semantic segmentation. Also, this is a many-to-many mapping, which means that for a given face there are multiple valid cartoons, and for a given cartoon there are multiple valid faces too.

Dataset

Faces dataset: we use the VggFace dataset (https://www.robots.ox.ac.uk/~vgg/data/vgg_face/) from the University of Oxford

Cartoon dataset: we use the CartoonSet dataset from Google (https://google.github.io/cartoonset/), both the versions of 10000 and 100000 items.

We filtered out the data just to keep realistic cartoons and faces images. This code is in scripts. To download the dataset:

1. pip3 install gdown
2. gdown https://drive.google.com/uc?id=1tfMW5vZ0aUFnl-fSYpWexoGRKGSQsStL
3. unzip datasets.zip

Directory structure

config.json: contains the model configuration to train the model

weights: contains weights that we saved the last time we trained the model.

├── api.py
├── config.json
├── images
│   ├── Cartoons_example.jpeg
│   └── Faces_example.jpeg
├── LICENSE
├── losses
│   └── __init__.py
├── models
│   ├── avatar_generator_model.py
│   ├── cdann.py
│   ├── decoder.py
│   ├── denoiser.py
│   ├── discriminator.py
│   ├── encoder.py
│   └── __init__.py
├── README.md
├── requirements.txt
├── scripts
│   ├── copyFiles.sh
│   ├── download_faces.py
│   ├── keepFiles.sh
│   ├── plot_utils.py
│   └── preprocessing_cartoons_data.py
├── train.py
├── utils
│   └── __init__.py
└── weights

The model

Our codebase is in Python3. We suggest creating a new virtual environment.

• The required packages can be installed by running pip3 install -r requirements.txt
• Update N_CUDA by running export N_CUDA=<gpu_number> if you want to specify the GPU to use

It is based on the XGAN paper omitting the Teacher Loss and adding an autoencoder in the end. The latter was trained to learn well only the representation of the cartoons as to "denoise" the spots and wrong colorisation from the face-to-cartoon outputs of the XGAN.

The model was trained using the hyperparameters located in config.json:

1. Change root_path in config json. It specifies where is datasets which contains the datasets.
2. Run python3 train.py --no-wandb
3. To launch tensorboard: tensorboard --logdir=<tensorboard-dir>

REST API

The codebase contains REST API endpoint for testing the model:

1. Update model_path in config.json to point to your model
2. Run api.py
3. POST image file to 0.0.0.0:9999/generate as a form parameter with name image