Facial expression recognition through adaptive learning of local motion descriptor

Abstract

A novel bag-of-words based approach is proposed for recognizing facial expressions corresponding to each of the six basic prototypic emotions from a video sequence. Each video sequence is represented as a specific combination of local (in spatio-temporal scale) motion patterns. These local motion patterns are captured in motion descriptors (MDs) which are unique combinations of optical flow and image gradient. These MDs can be compared to the words in the bag-of-words setting. Generally, the key-words in the wordbook as reported in the literature, are rigid, i.e., are taken as it is from the training data and cannot generalize well. We propose a novel adaptive learning technique for the key-words. The adapted key-MDs better represent the local motion patterns of the videos and generalize well to the unseen data and thus give better expression recognition accuracy. To test the efficiency of the proposed approach, we have experimented extensively on three well known datasets. We have also compared the results with existing state-of-the-art expression descriptors. Our method gives better accuracy. The proposed approach have been able to reduce the training time including the time for feature-extraction more than nine times and test time more than twice as compared to current state-of-the-art descriptor.

Appendix A: Convergence of the Adaptive Learning

Suppose that in the ith iteration of the while loop of Algorithm 1, ξ number of MDs (Q ₁,Q ₂,...,Q _ξ ) find key-MD P _l as their nearest key-MD. For illustration purpose, a typical positioning of the ξ number of MDs and the key-MD P _l in 2D space is shown in Fig. 14. Let the distance between the key-MD P _l and MDs Q _k , k=1,2,...,ξ be represented by d _k , k=1,2,...,ξ and $S={\sum }_{k=1}^{\xi }{d_{k}}$, i.e., S represents the summation of the distances from the key-MD P _l to the MDs Q _k , k=1,2,3,...,ξ. Without loss of generality, let us consider the MD Q ₁. After moving P _l by a fraction (δ) of the distance d ₁ towards Q ₁, let the new position of the key-MD be ${P^{1}_{l}}$. Let the new distances between the key-MD ${P^{1}_{l}}$ and MDs Q _k be represented by ${d^{1}_{k}}$ and let, $S^{1}={\sum }_{k=1}^{\xi }{{d^{1}_{k}}}$. We want S to decrease with each iteration. Let the angles between the straight lines joining Q _k to P _l and P _l to ${P^{1}_{l}}$ be 𝜃 _k and $\lambda ^{1}=d_{1} - {d^{1}_{l}}$. Therefore, ${d^{1}_{k}}-d_{k}=\sqrt {((d_{k})^{2}+(\lambda ^{1})^{2}-2d_{k}\lambda ^{1}\cos \theta _{k})}-d_{k}$. It can be shown that,

$$ \frac{dS^{1}}{d\lambda^{1}}=\frac{d}{d\lambda^{1}}\{\sum\limits_{k=1}^{\xi}{{d^{1}_{k}}}\} =-\sum\limits_{k=1}^{\xi}\cos\theta_{k} =-1-\sum\limits_{k=2}^{\xi}\cos\theta_{k}. $$

(7)

Fig. 14

A typical positioning of the ξ number of MDs, the key-MD P _l and its new position ${P^{1}_{l}}$ in 2D space

From (7) it can be said that rate of change of S ¹ with respect to λ ¹ is negative i.e., the key-MD converges towards the corresponding MDs, when ${\sum }_{k=2}^{\xi }\cos \theta _{k}>-1\text {.}$

Let ζ _k,j , k,j=1,2,...,ξ represent the distance between the two MDs Q _j and Q _k . Therefore, from (7) we can write, ${d^{1}_{k}} < d_{k}$ when (d _k )²+(d ₁)²>ζ _k,1. Similar equations can be inferred when we consider all the other MDs Q _k , k=2,...,ξ. Therefore, it can be said that the key-MD P _l converges towards the cluster of MDs Q _k , k=1,2,...,ξ as long as P _l remains outside the hyper circle whose diameter is the distance between the two maximum distant MDs among all Q _k , k=1,2,...,ξ. We find the lower bound of S−S ^ξ where S ^ξ is the summation of distances from the key-MD P _l to all the MDs Q _k , k=1,2,...,ξ after one full iteration of the while loop of Algorithm 1. From Fig. 14 we get (from the triangle inequality),

$$\begin{array}{@{}rcl@{}} &&\sum\limits_{k=2}^{\xi}{{d^{1}_{k}}} + ({\xi}-1){d^{1}_{1}} \leq \sum\limits_{k=2}^{\xi}{d_{k}} + ({\xi}-1)d_{1} \\ &\Rightarrow &\sum\limits_{k=1}^{\xi}{d_{k}} - \sum\limits_{k=1}^{\xi}{{d^{1}_{k}}} \geq ({\xi}-2)\left( {d^{1}_{1}}-d_{1}\right)\text{.}\\ &\text{Similarly,}& \sum\limits_{k=1}^{\xi}{{d^{1}_{k}}} - \sum\limits_{k=1}^{\xi}{{d^{2}_{k}}} \geq ({\xi}-2)\left( {d^{2}_{2}}-{d^{1}_{2}}\right)\\ &&.\\ &&.\\ &&.\\ &&\sum\limits_{k=1}^{\xi}{d^{{\xi}-1}_{k}} - \sum\limits_{k=1}^{\xi}{d^{{\xi}}_{k}} \geq ({\xi}-2)\left( d^{\xi}_{\xi}-d^{{\xi}-1}_{\xi}\right). \end{array} $$

(8)

Summing up the above ξ number of inequations we get,

$$\begin{array}{@{}rcl@{}} &&\sum\limits_{k=1}^{\xi}{d_{k}} - \sum\limits_{k=1}^{\xi}{d^{\xi}_{k}} \geq ({\xi}-2)\left( {d^{1}_{1}}-d_{1}+{d^{2}_{2}}-{d^{1}_{2}}+ ... + d^{\xi}_{\xi}-d^{\xi-1}_{\xi}\right) \\ &&\hspace*{20pt}\Rightarrow S-S^{\xi} \geq -({\xi}-2)(\lambda^{1} + \lambda^{2} + ... + \lambda^{\xi}). \end{array} $$

(9)

Since λ ^k, k=1,2,...,ξ are assumed to be very small, the magnitude of the right side of (9) is very small. We find the value of P _l for which S ^ξ gets its minimum value as follows. Since, S ^ξ is the summation of some distances, it is a positive number. Therefore, S ^ξ gets its minimum value for the value of P _l for which ${\sum }_{k=1}^{\xi }{(Q_{k}-P_{l})^{2}}$ gets its minimum value. Let us define $S^{\prime }={\sum }_{k=1}^{\xi }{(Q_{k}-P_{l})^{2}}$. Therefore,

$$\begin{array}{@{}rcl@{}} dS^{\prime} &=& -2\sum\limits_{k=1}^{\xi}{(Q_{k}-P_{l})dP_{l}} =-2\xi\{(\sum\limits_{k=1}^{\xi}{(Q_{k}}/\xi)-P_{l})\}dP_{l} \\ &=& -2\xi(\bar{Q}-P_{l})dP_{l} \end{array} $$

(10)

In (10), $\bar {Q}$ refers to the arithmetic mean of the MDs Q _k , k=1,2,...,ξ. From (10) it can be said that $dS^{\prime }$ and therefore $S^{\prime }$ gets its minimum value when $dP_{l} =\bar {Q}-P_{l}$ or $P_{l}=\bar {Q}$. The characteristic and the biggest advantage of adaptive learning technique is that the system under adaptive learning changes gracefully according to the changes in its environment. That is the key-MD under adaptive learning stores the experience from past learning. This is not possible if we consider $\bar {Q}$ to be the key-MD. Therefore, we represent the video sequences (training and test) in terms of ED constructed using adapted key-MDs.