Facial Expression Recognition Software

13.01.2020

Applying the facial expression recognition algorithm, the developed prototype is capable of processing a sequence of frontal images of moving faces and recognizing the person’s facial expression. Amazon Rekognition is a simple and easy to use API that can quickly analyze any image or video file stored in Amazon S3. Amazon Rekognition is always learning from new data, and we are continually adding new labels and facial recognition features to the service.

Facial Recognition Software For Photos
Facial Expression Recognition Software Free Download
Facial Expression

From Luigi Rosa:

Facial expression recognition - Matlab Source Code. We propose an algorithm for facial expression recognition which can classify the given image into one of the seven basic facial expression categories (happiness, sadness, fear, surprise, anger, disgust and neutral). This approach results extremely powerful because it does not require the detection of any reference point or node grid. The proposed method is fast and can be used for real-time applications.

What do you need to know about free software?

From Luigi Rosa:

Free print shop online. Using a Convolutional Neural Network (CNN) to recognize facial expressions from images or video/camera stream.

MobilEdit Lite introduces you to the functions of the full-featured paid app, and the trial limitations it comes with include the ability to read data but not write it, and the option to make up. Each application is listed and the newest version is able to be downloaded from this page. Includes applications and phone drivers for download. A link to What's New for each program is also on the page.

The goal is to get a quick baseline to compare if the CNN architecture performs better when it uses only the raw pixels of images for training, or if it's better to feed some extra information to the CNN (such as face landmarks or HOG features). The results show that the extra information helps the CNN to perform better.

To train the model, we used Fer2013 datset that contains 30,000 images of facial expressions grouped in seven categories: Angry, Disgust, Fear, Happy, Sad, Surprise and Neutral.

The faces are first detected using opencv, then we extract the face landmarks using dlib. We also extracted the HOG features and we input the raw image data with the face landmarks+hog into a convolutional neural network.

For our experiments, we used 2 CNN models:

Fer2013 is a challenging dataset. The images are not aligned and some of them are uncorrectly labeled as we can see from the following images. Moreover, some samples do not contain faces.

This makes the classification harder because the model have to generalize well and be robust to incorrect data. The best accuracy results obtained on this dataset, as far as I know, is 75.2% described in this paper:[Facial Expression Recognition using Convolutional Neural Networks: State of the Art, Pramerdorfer & al. 2016]

Experiments	SVM	Model A	Model B	Difference
CNN (on raw pixels)	-----	72.4%	73.5%	+1.1%
CNN + Face landmarks	46.9%	73.5%	74.4%	+0.9%
CNN + Face landmarks + HOG	55.0%	68.7%	73.2%	+4.5%
CNN + Face landmarks + HOG + sliding window	59.4%	71.4%	75.1%	+3.7%

As expected:

The CNN models gives better results than the SVM (You can find the code for the SVM implmentation in the following repository: Facial Expressions Recognition using SVM)
Combining more features such as Face Landmarks and HOG, improves slightly the accuray.
Since the CNN Model B uses deep convolutions, it gives better results on all experiments (up to 4.5%).

It's interesting to note that using HOG features in the CNN Model A decreased the results compared to using only the RAW data. This may be caused by an overfitting or a failure to extract the coorelation between the information.

In the following table, we can see the effects of the batch normalization on improving the results:

Batch norm effects	on Model A	on Model B
CNN (on raw pixels)	+7.4%	+39.3%
CNN + Face landmarks	+26.2%	+50.0%
CNN + Face landmarks + HOG	+1.9%	+50.1%
CNN + Face landmarks + HOG + sliding window	+16.7%	+16.9%

In the previous experiments, I used only 5 expressions for the training: Angry, Happy, Sad, Surprise and Neutral.

The accuracy using the best model trained on the whole dataset (7 emotions) dropped to 61.4%.The state of the art results obtained on this dataset, as far as I know, is 75.2% described in this paper.

Note: the code was tested in python 2.7 and 3.6.

Tensorflow
Tflearn
Numpy
Argparse
[optional] Hyperopt + pymongo + networkx
[optional] dlib, imutils, opencv 3
[optional] scipy, pandas, skimage

Better to use anaconda environemnt to easily install the dependencies (especially opencv and dlib)

Facial Recognition Software For Photos

Download Fer2013 dataset and the Face Landmarks model
Unzip the downloaded files
And put the files fer2013.csv and shape_predictor_68_face_landmarks.dat in the root folder of this package.
Convert the dataset to extract Face Landmarks and HOG Features
You can also use these optional arguments according to your needs:
- -j, --jpg (yes no): save images as .jpg files (default=no)
- -l, --landmarks(yes no): extract Dlib Face landmarks (default=yes)
- -ho, --hog (yes no): extract HOG features (default=yes)
- -hw, --hog_windows (yes no): extract HOG features using a sliding window (default=yes)
- -hi, --hog_images (yes no): extract HOG images (default=no)
- -o, --onehot (yes no): one hot encoding (default=yes)
- -e, --expressions (list of numbers): choose the faciale expression you want to use: 0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral (default=0,1,2,3,4,5,6)
Examples:
The script will create a folder with the data prepared and saved as numpy arrays.Make sure the --onehot argument set to yes (default value)

Choose your parameters in 'parameters.py'
Launch training:

The variable output_size in parameters.py (line 20), should correspond to the number of facial expressions you want to train on. By default it is set to 7 expressions.

Train and evaluate:

N.B: make sure the parameter 'save_model' (in parameters.py) is set to True if you want to train and evaluate

For this section, you'll need to install first these optional dependencies:

Lunch the hyperparamets search:

You should then retrain your model with the best parameters

N.B: the accuracies displayed are for validation_set only (not test_set)

Modify 'parameters.py':

Set 'save_model_path' parameter to the path of your pretrained file.

Launch evaluation on test_set:

For this section you will need to install dlib and opencv 3 dependencies
Modify 'parameters.py':

Set 'save_model_path' parameter to the path of your pretrained file

Predict emotions from a file

For this section you will need to install dlib, imutils and opencv 3 dependencies
Modify 'parameters.py':

Set 'save_model_path' parameter to the path of your pretrained file

Predict emotions from a file

A window will appear with a box around the face and the predicted expression.Press 'q' key to stop.

N.B: If you changed the number of expressions while training the model (default 7 expressions), please update the emotions array in parameters.py line 51.