Spontaneous micro-expression spotting via geometric deformation modeling

Highlights

• A probabilistic framework is proposed to detect spontaneous micro-expression clips.

• The geometric deformation captured by ASM model is utilized as features.

• The features are robust to subtle head movement and illumination variation.

• The Adaboost algorithm is used to estimate the initial probability for each frame.

• The random walk algorithm computes the transition probability by deformation similarity.

• Extensive experiments are performed on two spontaneous datasets.

Abstract

Facial micro-expression is important and prevalent as it reveals the actual emotion of humans. Especially, the automated micro-expression analysis substituted for humans begins to gain the attention recently. However, largely unsolved problems of detecting micro-expressions for subsequent analysis need to be addressed sequentially, such as subtle head movements and unconstrained lighting conditions. To face these challenges, we propose a probabilistic framework to detect spontaneous micro-expression clips temporally from a video sequence (micro-expression spotting) in this paper. In the probabilistic framework, a random walk model is presented to calculate the probability of individual frames having micro-expressions. The Adaboost model is utilized to estimate the initial probability for each frame and the correlation between frames would be considered into the random walk model. The active shape model and Procrustes analysis, which are robust to the head movement and lighting variation, are used to describe the geometric shape of human face. Then the geometric deformation would be modeled and used for Adaboost training. Through performing the experiments on two spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression spotting approach.

Introduction

Emotion exists in humans’ life, which can be revealed by external behaviors, such as vocal expression, facial expression, and sign expression. Among these behaviors, the facial expression plays a vital role in analyzing human emotions [1]. Therefore, the facial expression has attracted much attention in psychological studies [2]. Besides, with the development of computer science, the facial expression analysis and recognition become popular in fields of computer vision and pattern recognition [3]. Various machine learning techniques have been employed to automatically analyze and recognize facial expression from visual images or videos, such as local binary pattern (LBP) [4] and hidden Markov model (HMM) [5]. Most of these works are devoted to the macro-expression while the micro-expression manifests more affective information [6].

In contrast to macro-expression, the micro-expression is a brief, involuntary facial expression shown on the face of humans, which usually sustains from 1/25 to 1/5 s and has a period of onset, apex, and offset [7]. The psychological studies have shown the importance of micro-expression revealing the suppressed affect of humans, which helps to understand humans’ deceitful behaviors. Consequently, there are large amounts of areas to apply the micro-expression analysis and recognition, such as lie detection, police case diagnosis, business negotiation, psychoanalyzing, and so on. Due to the short duration and involuntariness of micro-expressions, it is very difficult for untrained people to detect and analyze micro-expressions. Even trained by professional tools, such as the micro expression training tool (METT) [8], numerous works might be accomplished manually by professionals to detect and analyze micro-expressions from videos. Therefore, the automated detection and analysis of micro-expressions would be very valuable and help people promote the performance of analyzing large amounts of video sequences.

The automated micro-expression detection from temporal video sequences attracts few attentions while some works have been devoted to the micro-expression recognition based on well-segmented video sequences containing the micro-expressions. Although the micro-expression detection is more fundamental to subsequent micro-expression analysis and recognition, however, few works have been presented to detect micro-expressions [9], [10]. To apply the detection in real-life scenarios, several problems of micro-expression context would be addressed. The very small head movements and heterogeneous ambient lighting conditions induce the context complexity. So the head movements and lighting variation have potentially significant effects on subtle changes of micro-expressions. To face these challenges, we present a framework to detect consecutive frame clips having micro-expressions from video sequences, which would be robust for small head movement and lighting variation. In addition, the deliberate micro-expressions differ greatly from the spontaneous ones as they are controlled by different motor pathways [11]. Since spontaneous micro-expressions can be observed frequently in real life and reveal more affective information of humans, we focus on the problem of detecting spontaneous micro-expression frame clips temporally from video sequences, which is called as spontaneous micro-expression spotting in this context.

To be dedicated to the problem of micro-expression spotting, we propose a random walk framework to detect frame clips having micro-expressions in video sequences. In the probabilistic model, the main contributions of our proposed approach are summarized as follows

• The random walk (RM) model is applied to compute the probability of frames having micro-expressions. The model can leverage the deformation correlation between frames and spot the consecutive frame clips with micro-expressions.

• The Adaboost algorithm is utilized to compute the initial probability of individual frames having micro-expressions in the RW procedure. The thresholding weak classifiers have been trained for obtaining the best geometric features for micro-expression spotting.

• In order to prevent influences of the head movement and lighting variation, a revised active shape model (ASM) is used to locate the landmarks and then the Procrustes analysis is presented to align these landmarks. The geometric deformations of landmarks are modeled as the features of classifier training and used to compute the transition probability in RW model.

The rest of this paper is organized as follows. Section 2 reviews the related work briefly and our proposed probabilistic framework for micro-expression spotting is presented in Section 3. Then we discuss the experimental results for algorithm evaluation in Section 4. Finally, Section 5 describes our conclusions.