Face Preparation for Model Prediction

[lukkowal5]

Hi,

How can I read/align/prepare face for making predict on trained model?
I made code like this:

    model = create_deepface()
    model.load_weights('VGGFace2_DeepFace_weights_val-0.9034.h5')
    img = load_img(path)
    img = img.resize((152,152))
    img_array = img_to_array(img)
    img_array = img_array.reshape((-1, 152, 152, 3))
    res = model.predict(img_array)

I get different results for every face belonging to the same person.

Thanks a lot for a help

[swghosh]

You need to use the preprocess_image function which is an essential part of the train and test data pipeline. In your case the function can be called in the following way.

# required imports
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from deepface import create_deepface
from deepface import Dataset

model = create_deepface()
model.load_weights('VGGFace2_DeepFace_weights_val-0.9034.h5')

# fine tune the deepface model for your dataset
# for fine tuning, remove last dense layer only; (as per paper)
# set all pretrained parameters to non trainable
# add a dense layer with dim = num_classes (your dataset)
# set dense layer to trainable and train the model

img = load_img(path)
img = img.resize((152,152))
img_array = img_to_array(img)
img_array = img_array.reshape((-1, 152, 152, 3))
res = model.predict(img_array)

If there are any issues regarding the fine tuning step, please do let me know.

[swghosh]

For training the DeepFace network, 2D aligned (2D frontalisation using 5 fiducial points similar to the one specified in paper) images from the VGGFace2 dataset have been used. It is expected that we use the same alignment technique (2D alignment of tight face crops) to make accurate predictions when using DeepFace feature vectors.

Shortly, will update the repository with the code required to obtain 2D aligned tight face crops from images in dataset. 2D-alignment in our case had been achieved using dlib.get_face_chips method using shape_predictor_5_face_landmarks.

Currently, I'm working on the 3D frontalisation step to replicate the exact 3D alignment step as specified in paper. Hope to add include those changes and train the network using 3D frontalised images soon.

[lukkowal5]

Thank you 👍

Now I have code like this:

from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import array_to_img
import numpy as np
import dlib;
from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image

detector = dlib.get_frontal_face_detector()
sp = dlib.shape_predictor('shape_predictor_5_face_landmarks.dat')

def processdeepface(path):
    img = dlib.load_rgb_image(path)
    dets = detector(img, 1)
    for k, d in enumerate(dets):
        shape = sp(img, d)
        face_chip = dlib.get_face_chip(img, shape, size=152)
        image = preprocess_image(face_chip)
        img_array = img_to_array(image)
        img_array = img_array.reshape((1, IMAGE_SIZE[0], IMAGE_SIZE[1], 3))
        res = model.predict(img_array)

Am I doing it in proper way?
Faces belonging to the same person should have the same output up? Am I correct?

Thanks

[swghosh]

The code for the 2D face alignment seems fine.
The only limitation being that the model.predict method can make accurate predictions only after the model has been fine tuned (last dense layer only) on your required dataset.

The original implementation (which actually represents the pretrained weights) makes use of a softmax classifier trained on 8631 classes from the VGGFace2 dataset. So, it is unlikely to make predictions the same class label unless the face crop being used belongs to one of those classes.

DeepFace doesn't use a triplet loss based implementation unlike FaceNet by Scroff et al. which can be directly used for a face verification task. Instead, the DeepFace feature vector obtained from the second last dense layer from the network can be effectively used to learn a face verification problem.

[ZHUANG-JLU]

Thank you

Now I have code like this:

from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import array_to_img
import numpy as np
import dlib;
from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image

detector = dlib.get_frontal_face_detector()
sp = dlib.shape_predictor('shape_predictor_5_face_landmarks.dat')

def processdeepface(path):
    img = dlib.load_rgb_image(path)
    dets = detector(img, 1)
    for k, d in enumerate(dets):
        shape = sp(img, d)
        face_chip = dlib.get_face_chip(img, shape, size=152)
        image = preprocess_image(face_chip)
        img_array = img_to_array(image)
        img_array = img_array.reshape((1, IMAGE_SIZE[0], IMAGE_SIZE[1], 3))
        res = model.predict(img_array)

Am I doing it in proper way?
Faces belonging to the same person should have the same output up? Am I correct?

Thanks

@swghosh @lukkowal5 Hi,how can i process the 8631 output to a image? Thanks.

[swghosh]

@ZHUANG-JLU
Hello,
Shortly I'm sharing a complete piece of code which'll help you perform face verification based on the DeepFace pre-trained model.

Sorry for the delay in reply.

[swghosh]

@ZHUANG-JLU,
What is the actual problem that you're motivated to solve?

Is it a face verification task (relating to matching a pair of images whether they belong to same person or not) or a face recognition task (relating to a multi class classification problem with some k number of people/identities in dataset)?

FYI, the last layer of the CNN i.e. the F8 layer is trained to classify samples of the VGGFace2 dataset in this case. The pre-final layer, F7 layer with 4096 dimensions is used to extract the DeepFace feature vector. The simplest workaround for a face recognition problem is to replace the F8 layer with a new dense layer of dimension equal to number of classes in new dataset, and retrain the network for a few epochs (eg. 20) with all weights frozen upto F7. [some similar work to this is available here: FineTuning DeepFace on a smaller dataset.ipynb]

Please confirm about the same then I can get back to you with some code that can perform either based on your requirement.

[ZHUANG-JLU]

@ZHUANG-JLU,
What is the actual problem that you're motivated to solve?

Is it a face verification task (relating to matching a pair of images whether they belong to same person or not) or a face recognition task (relating to a multi class classification problem with some k number of people/identities in dataset)?

FYI, the last layer of the CNN i.e. the F8 layer is trained to classify samples of the VGGFace2 dataset in this case. The pre-final layer, F7 layer with 4096 dimensions is used to extract the DeepFace feature vector. The simplest workaround for a face recognition problem is to replace the F8 layer with a new dense layer of dimension equal to number of classes in new dataset, and retrain the network for a few epochs (eg. 20) with all weights frozen upto F7. [some similar work to this is available here: FineTuning DeepFace on a smaller dataset.ipynb]

Please confirm about the same then I can get back to you with some code that can perform either based on your requirement.

Face Detection/Recognition/Alignments.Thanks for your reply.

[swghosh]

@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb

[ZHUANG-JLU]

@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb

I will see it.Thanks.

[thepranaygaur]

Hey there, I am new in this field can you tell me how to run this code.

[thepranaygaur]

Also is there any way I can run it on my custom dataset.

[swghosh]

@ZHUANG-JLU,
Did you look into this code?
FineTuning DeepFace on a smaller dataset.ipynb

Hi @thepranaygaur,

Can you refer to the notebook provided in the link to run the face recognition model on a custom dataset?

Assuming that you've a directory containing pictures of faces of people (tight face crops, aligned) segregated into sub directories basis their class labels / identity of the person, you can use the notebook for end to end code.

Do let me know if you have any queries.

For face detection and alignment please use dlib.get_frontal_face_detector and dlib.get_face_chips respectively as given in #1 (comment)

[FunkyKoki]

Hello, thanks for your work first.
I am reading DeepFace, and I am quite confused about the 3D alignment step. To be specific, I cannot tell what is the affine 3D-to-2D camera P (what are the 8 unknowns mean?). Please help me. Thank you a lot.

[swghosh]

Hi @FunkyKoki, thanks for reaching out!

In my repository, the DeepFace model which have been trained on VGGFace2 dataset uses tight face crops with 2D alignment only (atleast, as of now) using dlib.get_face_chips (works reasonably well under various poses, except poses that lack frontal face which is the reason I had remove from training the face pictures that lacked frontal poses). I went through the 3D alignment process as discussed in the original paper but failed to replicate the work primarily due to lack of publicly available 3D face datasets and level of difficulty associated with it.

The methodology discussed uses 3D to 2D mapping of face keypoints with the help of an affine camera (not really sure how that can be done with regular 2D face crops) and a trained 3D face model. As a starting point, I'd recommend you to get started with these resources which should help you develop more clarity on this topic.

• Viewing Real-World Faces in 3D (https://talhassner.github.io/home/publication/2013_ICCV)
• face-frontalization repo by dougsouza (https://github.com/dougsouza/face-frontalization) - this is a Python port of the original MATLAB implementation of the paper Effective Face Frontalization in Unconstrained Images (https://talhassner.github.io/home/publication/2015_CVPR_1)
  Aforementioned, includes work done by Tal Hassner who has relevant work in this field (also, cited in DeepFace paper).
• Introductory details about camera models and 3D affine cameras (https://www-users.cs.umn.edu/~hspark/CSci5980/CameraModel1.pdf)

Let me know if you are willing to collaborate with me on this at a suitable time / looking forward to contributing some code that is related to 3D face frontalization and related model training. Will be more than happy to be of any help.

Good luck!

[FunkyKoki]

Thank you so much for your sharing.
Indeed, I have 2 questions about the 3D-to-2D process.
First, what are the 8 unknows in the affine camera P mean? Location, gesture and lens?
Second, the author added the residual r into the 3d coordinates, but this operation can be done only if the coordinates systems are aligned, how could that be done?

I would check the resources you shared and try to re-implement the whole process. By the way, actually, in the modern network, the strong 3D alignment seems not to be a necessity. I am interested in the 3D alignment only because that I am working in the 3D reconstruction of face.

I would let you know when I solve the two questions above.

Thanks again.

[zzzzmoya]

@swghosh Hello, I'm trying to use https://github.com/serengil/deepface deepFace facial attribute analysis, but the performance toward certain age group is bad. So I'm looking for a way to train/ tune the age/gender/race classifier with my own dataset. I'm wondering if this is possible to do? I saw this repo is using VGG face for pretrain weight. Can I retrain with my dataset? May you give me some insight? Thank you!

[swghosh]

Hey @zzzzmoya,
That sounds like an interesting concept. Indeed, the original VGG-Face2 dataset focuses on a detailed study about various facial attributes. I'm not sure about what serengil/deepface offers with regard to Facial Attribute Analysis (but it does use the weights from my repo for what they call as FB-DeepFace). I'm skeptical that the original residual architecture mentioned (if I remember correctly) in the VGGFace2 paper should help you with get a head start on your work. And, I'd assume that the VGGFace2 dataset is inclusive enough irrespective of age groups so you can train your own model with it and later tune it on your dataset. Feel free to ping me if there's related additional discussion to be done on this regard (preferably, by creating a separate issue).

Unfortunately a no to using those weights for facial attribute analysis as I'm not sure if it's a great fit with the existing weights ATM. (as the weights are classification task specific; also I'm not sure about the effects of using the DeepFace architecture for such tasks)

I saw this repo is using VGG face for pretrain weight. Can I retrain with my dataset? May you give me some insight?

[rkdass]

Hey @swghosh, awesome post! I have a question:

Could the classification task be repurposed to say facial analysis? Eg: Using the same data hierarchical structure, what if instead of matching (one to many) subjects, the labels would be race-ethnicity identification where folders would be labeled into a limited number of categories?

From your reply to @ZHUANG-JLU, I'm thinking of creating a new F8 layer with the number of categories for my task and train the model first as a classifier (i.e. training just the F8 layer) using VGGFace2 pre-trained weights and then fine-tuning setting a portion of the DeepFace model to trainable. Curious to know your thoughts? Would you by any chance have any sample code for creating an additional F8 layer? Thanks!

[swghosh]

Hey @swghosh, awesome post! I have a question:

Could the classification task be repurposed to say facial analysis? Eg: Using the same data hierarchical structure, what if instead of matching (one to many) subjects, the labels would be race-ethnicity identification where folders would be labeled into a limited number of categories?

From your reply to @ZHUANG-JLU, I'm thinking of creating a new F8 layer with the number of categories for my task and train the model first as a classifier (i.e. training just the F8 layer) using VGGFace2 pre-trained weights and then fine-tuning setting a portion of the DeepFace model to trainable. Curious to know your thoughts? Would you by any chance have any sample code for creating an additional F8 layer? Thanks!

1. I am certain that the output of F7 layer can be used as features for other facial related tasks as well, Facial Analysis in your case. Yes, there are quite a lot of chances that adding a new layer on top F7 or even experimenting the same with the output of L6 and then stacking custom Dense layers should help you fine-tune the model on a new facial task with new categories.
   In case you want a model that performs generic facial analysis, you should be able to do that using a MTCNN (multi-task CNN) which would provide for multi-label classification as well as regression tasks under a common end-to-end neural network.
   For just race/ethnicity classification, I'd hope that output of features from either F7 or L6 (can try experimenting with either: F7 preferred) should suffice given there is one or more Dense layer(s) stacked on top (subject to hyper-parameter tuning and experiments).

2. Would this notebook not be sufficient for you to guide you on training a new face model with new classification categories? (in your cases, categories would be race/ethnicity labels; everything else code-wise should be the same).
   FineTuning DeepFace on a smaller dataset.ipynb
   You could also refer to any resources where generic fine-tuning tasks using Keras models have been described for that purpose. Should be pretty easy to load the weights and just remove the last layer to use your own.
   As an extra, if you'd like to fine-tune the complete model (inclusive of Conv layers, Locally-connected layers) but please note that it is extremely computationally expensive and would either need a GPU cluster or a TPU. (Google Colab provides for free TPUs which should suffice)

[saidafef]

hi ,
i used FineTuning DeepFace on a smaller dataset.ipynb with my dataset but i can't get good accuracy and it give me just 10%
thnx for help

[swghosh]

Hi @saidafef: Apologies for the delayed response.

You could probably check for any red flags in any of the following aspects:

1. Pre-processing is same as that used in training or not
2. The size of the dataset and number of samples per class should be reasonable as the DeepFace feature classifier would perform only as good as the number of samples it is trained on for each class (unlike one-shot face verification tasks like FaceNet)
3. Occlusions / necessary variations among your training and validation datasets
4. Learning rate can be tuned significantly along with dropout value in range 0.2-0.5 for some performance improvements
5. Full fine tuning with a very low exponentially decayed learning rate? (only you have cumbersome resources available because training the locally connected layers is too much resource hungry)

From my personal standpoint I have been able to achieve 95+% accuracy on the test by training it on a custom face dataset created from scratch, with very different set of occlusions on the train and testing sets. (training set had very less variety in terms of occlusions/protrusions whereas the test set had a lot of them and were quite different in terms of lighting as well; suprisingly the features transfer learned from DeepFace were able to capture correlation in facial structure giving good results even on a very different test set)

Feel free to ping with any additional details.

[gautamHCSCV]

Hi @swghosh , I am looking for a notebook of face verification problem (given two images identify if they are of the same individual or of different people). Do you have a similar implementation like this (https://colab.research.google.com/drive/1otl-4tJaCfyCJF33bvnjzeuUdldcWysA) for my case. We want to fine tune the model to identify images of same people in a given dataset.