Facial expression recognition via learning deep sparse autoencoders

Abstract

Facial expression recognition is an important research issue in the pattern recognition field. In this paper, we intend to present a novel framework for facial expression recognition to automatically distinguish the expressions with high accuracy. Especially, a high-dimensional feature composed by the combination of the facial geometric and appearance features is introduced to the facial expression recognition due to its containing the accurate and comprehensive information of emotions. Furthermore, the deep sparse autoencoders (DSAE) are established to recognize the facial expressions with high accuracy by learning robust and discriminative features from the data. The experiment results indicate that the presented framework can achieve a high recognition accuracy of 95.79% on the extended Cohn–Kanade (CK+) database for seven facial expressions, which outperforms the other three state-of-the-art methods by as much as 3.17%, 4.09% and 7.41%, respectively. In particular, the presented approach is also applied to recognize eight facial expressions (including the neutral) and it provides a satisfactory recognition accuracy, which successfully demonstrates the feasibility and effectiveness of the approach in this paper.

Introduction

Facial expression, as one of the most significant means for human beings to show their emotions and intensions in the process of communication, plays a significant role in human interfaces. In recent years, facial expression recognition has been under especially intensive investigation, due conceivably to its vital applications in various fields including virtual reality, intelligent tutoring system, health-care and data driven animation [1], [5], [12], [46]. The main target of facial expression recognition is to identify the human emotional state (e.g., anger, contempt, disgust, fear, happiness, sadness, and surprise [11]) based on the given facial images. It should be pointed out that it is a challenging task to automatically recognize facial expressions with high accuracy. On one hand, it is difficult to find the similarity of the same emotion state between different persons since they may express the same emotion state in various ways. On the other hand, it is also hard to seek the difference between expressions of the same person because some emotion states are too subtle to discriminate. Nevertheless, several approaches have been proposed to automatically recognize facial expressions. Generally, these methods can be separated into two classifications: the feature-based approaches and the template-based approaches [12], [27].

In this paper, we concentrate on the feature-based approach, where the expression information are extracted from appearance or geometrical features [28], [34], [35]. Here, geometric features denote the locations and shape of facial components, and the appearance features express the facial appearance changes, such as furrows, gapes, wrinkles, and bulges. Especially, the most important procedure for facial expression recognition is to extract representative features from original facial images so as to successfully distinguish different emotions. Obviously, the combination of geometric and appearance features can provide more effective facial representation because the collected features include not only the exact locations but also the skin changes. In addition, a high-dimensional feature for face recognition has been proved effective, and the performance is superior to both low-dimensional feature and the-state-of-art in most cases [3]. Inspired by this idea, we introduce the high-dimensional feature into facial expression recognition with hope to present a novel and powerful method. Up to now, a variety of statistical features have been put forward, as well as applied to the expression detection, such as local binary patterns (LBP) [31], scale invariant feature transformation (SIFT) [17] and Gabor filters [33], [49]. Particulary, the histogram of oriented gradient (HOG) [7], [13], as a good descriptor of the local appearance and also the shape, has been exploited to expression analysis in recent years. In this paper, we first locate the accurate position of dense facial landmarks with face alignment method. After that, the high-dimensional feature is formed by concatenating all descriptors which extracted from patches centered around landmarks. Specifically, three different descriptors, which are HOG, LBP and gray value, are selected and evaluated in this paper.

Deep sparse autoencoders (DSAE), one of the deep learning models, have been extensively researched and widely applied to many fields [14], [37]. In particular, the DSAE is a deep neural network built by stacked sparse Autoencoders, and the Softmax classifier is generally selected as the output layer for classification problem [32], [36]. The highlight of DSAE is that it can extract useful features by unsupervised learning, that is, it only reserves crucial information of the data in robust and discriminative representations after detecting and removing input redundancies [26]. It should be mentioned that it is a challenging task to distinguish different emotions regardless of the identity of the face because of the individual variations for the same expression and the subtleties between expressions. In addition, the external factors increase the difficulty of the recognition process, in terms of illumination, environment and cameras. So as to conquer the challenges mentioned above, we intend to establish a DSAE-based deep learning framework for facial expression recognition to classify the expressions with high accuracy by learning the useful features from the data set.

The novelty and contribution of our work are primarily threefold. (1) A high-dimensional feature, which is the combination of the facial geometric and appearance features, is introduced to the facial expression recognition due to its containing the accurate and comprehensive information of emotions. (2) A DSAE-based deep learning framework is established for facial expression recognition with high accuracy by learning robust and discriminative features from the data set. (3) The presented DSAE-based approach is successfully applied to distinguish different facial expressions on the CK+ database. Note that our work focuses on the 7-class and 8-class (including the neural) facial expression recognition, which are more difficult to distinguish. The results showed that the DSAE-based approach outperforms other three state-of-the-art approaches for 7-class recognition by as much as 3.17, 4.09, and 7.41 respectively, and it also achieves a good performance with satisfactory accuracy for 8-class recognition.

The remainder of this paper is organized as follows. In Section 2, we present a detailed introduction on the sparse autoencoder, the deep sparse autoencoders, as well as the applications to the facial expression recognition. Section 3 mainly discusses the experiment results of facial expression recognition via the deep sparse autoencoders and also evaluates its overall performance by comparing with other three state-of-the-art approaches. Finally, conclusions are summarized in Section 4.