Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projectionSignal Processing


Wei-Lun Chao, Jian-Jiun Ding, Jun-Zuo Liu
Control and Systems Engineering / Signal Processing / Electrical and Electronic Engineering / Software / Computer Vision and Pattern Recognition


Local binary pattern based features for sign language recognition

M. Hrúz, J. Trojanová, M. Železný

Regularization Preserving Localization of Close Edges

Olivier Laligant, Frdric Truchetet, Fabrice Meriaudeau

Agent-based recognition of facial expressions

Pablo Suau, Mar Pujol, Ramon Rizo, Simon Caton, Omer F. Rana, Bruce Batchelor, Francisco Pujol


Author's Accepted Manuscript

Facial expression recognition based on improved local binary pattern and classregularized locality preserving projection

Wei-Lun Chao, Jian-Jiun Ding, Jun-Zuo Liu

PII: S0165-1684(15)00142-5

DOI: http://dx.doi.org/10.1016/j.sigpro.2015.04.007

Reference: SIGPRO5784

To appear in: Signal Processing

Received date: 31 October 2014

Revised date: 9 April 2015

Accepted date: 10 April 2015

Cite this article as: Wei-Lun Chao, Jian-Jiun Ding, Jun-Zuo Liu, Facial expression recognition based on improved local binary pattern and classregularized locality preserving projection, Signal Processing, http://dx.doi.org/ 10.1016/j.sigpro.2015.04.007

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting galley proof before it is published in its final citable form.

Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. www.elsevier.com/locate/sigpro 1

Facial Expression Recognition Based on Improved

Local Binary Pattern and Class-regularized Locality

Preserving Projection

Wei-Lun Chao a, Jian-Jiun Ding * b, and Jun-Zuo Liu b a Department of Computer Science, University of Southern California, 3601 South Flower Street,

Tyler 1, Los Angeles, California 90089, U.S.A b Graduate Institute of Communication Engineering, National Taiwan University, No. 1, Sec. 4, Roosevelt Rd., Taipei, Taiwan, 10617, R.O.C. weilunchao@gmail.com, jjding@ntu.edu.tw, jinzuo2011@gmail.com,

TEL: 886-2-33669652, Fax: 886-2-33663662


This paper provides a novel method for facial expression recognition, which distinguishes itself with the following two main contributions. First, an improved facial feature, called the expression-specific local binary pattern (es-LBP), is presented by emphasizing the partial information of human faces on particular fiducial points. Second, to enhance the connection between facial features and expression classes, class-regularized locality preserving projection (cr-LPP) is proposed, which aims at maximizing the class independence and simultaneously preserving the local feature similarity via dimensionality reduction. Simulation results show that the proposed approach is very effective for facial expression recognition.

Keywords: Facial expression; Expression-specific local binary pattern; Class-regularized locality preserving projection; Dimensionality reduction; Feature extraction 2 1. Introduction

Because of its important role in human-computer interfaces (HCI), surveillance systems, and human entertainment, face recognition has attracted significant attention in pattern recognition and computer vision. Many algorithms about face verification [1, 2], facial age estimation [3], gender identification [4], and facial expression recognition [5-8] have been developed in recent years. In this paper, we focus on the problem of image-based facial expression recognition.

In general, algorithms of facial expression recognition can be simply divided into two steps: feature extraction and expression classification. In the first step, features that are related to the facial appearance or geometry [9-11] are extracted from the input face image for compact representation. Then, in the second step, according to the extracted features, an expression classifier [12, 13] is applied to assign the input face an expression label (e.g. seven expression labels—angry, disgust, fear, joy, sadness, surprise, and neutral, which are usually considered in the literature). Besides these two steps, some algorithms [14, 15] further include dimensionality reduction [16-18] as an intermediate step for avoiding the over-fitting problem and filtering out some irrelevant features on facial expression.

However, even with so much work as mentioned above, there is still a salient gap between human and machines’ ability on facial expression recognition. More accurately, some technical challenges have not been well solved. For example, different people, or even the same person, can have different expression patterns at different time or on different conditions. Therefore, how to design a robust feature extraction method that can handle this variety is still a critical problem in facial expression recognition. In addition, there intrinsically exist correlations among the seven expression classes, making some pairs of classes easy to be recognized; some others, hard to be classified. Nevertheless, the classifiers applied to facial expression recognition usually assume the independency among these classes: The widely-used support vector machine (SVM) [12] and K-nearest neighbor classifier (KNN) [19] consider no correlation among classes. This mismatch unavoidably leads to the bottleneck of expression recognition where some pairs of classes are always with low recognition rates.

Considering these two challenges, in this paper, we present an improved method for facial expression recognition, which is based on the conventional three-step framework and with the following 3 two main contributions:  Inspired by the human ability on expression recognition—with only partial information on faces, humans can still recognize facial expression with high accuracy—we propose the expression-specific local binary pattern (es-LBP) by computing the local LBP histograms [10] around some particular fiducial points of human faces. Besides, to further include the spatial information in each local LBP window, a symmetric extension for the es-LBP is also presented.  To alleviate the mismatch between the properties of expression classes and classifiers, we propose the class-regularized locality preserving projection (cr-LPP), which aims to push the samples of each class towards some pre-defined locations during the process of LPP [16], a popular dimensionality reduction algorithm in recent pattern recognition researches. Through specifically setting the pre-defined locations, the independence among classes can be enhanced, therefore effectively reducing the degree of mismatch.