A face emotion tree structure representation with probabilistic recursive neural network modeling

Abstract

This paper describes a novel structural approach to recognize the human facial features for emotion recognition. Conventionally, features extracted from facial images are represented by relatively poor representations, such as arrays or sequences, with a static data structure. In this study, we propose to extract facial expression features vectors as Localized Gabor Features (LGF) and then transform these feature vectors into FacE Emotion Tree Structures (FEETS) representation. It is an extension of the Human Face Tree Structures (HFTS) representation presented in (Cho and Wong in Lecture notes in computer science, pp 1245–1254, 2005). This facial representation is able to simulate as human perceiving the real human face and both the entities and relationship could contribute to the facial expression features. Moreover, a new structural connectionist architecture based on a probabilistic approach to adaptive processing of data structures is presented. The so-called probabilistic based recursive neural network (PRNN) model extended from Frasconi et al. (IEEE Trans Neural Netw 9:768–785, 1998) is developed to train and recognize human emotions by generalizing the FEETS representation. For empirical studies, we benchmarked our emotion recognition approach against other well known classifiers. Using the public domain databases, such as Japanese Female Facial Expression (JAFFE) (Lyons et al. in IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362, 1999; Lyons et al. in third IEEE international conference on automatic face and gesture recognition, 1998) database and Cohn–Kanade AU-Coded Facial Expression (CMU) Database (Cohn et al. in 7th European conference on facial expression measurement and meaning, 1997), our proposed system might obtain an accuracy of about 85–95% for subject-dependent and subject-independent conditions. Moreover, by testing images having artifacts, the proposed model significantly supports the robust capability to perform facial emotion recognition.

Appendix A

1.1 A.1. Proof of the condition 1 for PRNN convergence

By taking the expectation in (22) and as $E\{ {\mathbf{a}}_{k} (n)\Upphi_{j} \} = 0,$ we get

$$ E\{ {\mathbf{a}}_{k} (n + 1)\} = [{\mathbf{I}} - \eta {\mathbf{R}}_{\Upphi } \Upgamma ]E\{ {\mathbf{a}}_{k} (n)\} + \eta {\mathbf{r}}_{{{\text{d}}\Upphi }} \Upgamma , $$

(29)

where we can define:

$$ {\mathbf{R}}_{\Upphi } = E\left\{ {\Upphi_{j}^{R} \Upphi_{j}^{RT} } \right\}, $$

(30)

$$ {\mathbf{r}}_{d\Upphi } = E\left\{ {{\mathbf{d}}_{j} \Upphi_{{_{j} }}^{R} } \right\}, $$

(31)

and

$$ \Upgamma = E\left\{ {\frac{{\partial q^{ - 1} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }}} \right\}. $$

(32)

We refer to R _Φ as the correlation matrix of the input pattern set and to r _dΦ as the cross-correlation between the input pattern and the desired output. In the mean time, if we assume that $\Upgamma = E\left\{ {\Uplambda \left( {\Upphi_{j}^{1} } \right)\Upphi_{j}^{1} } \right\}$ as it is applied in a deep tree structure, so we can define a new matrix ${\mathbf{R}}_{\Upphi }^{*} = {\mathbf{R}}_{\Upphi } \Upgamma$ and ${\mathbf{r}}_{d\Upphi }^{*} = {\mathbf{r}}_{d\Upphi } \Upgamma$. In order to find the condition for the convergence in the mean, we can make use of an orthogonal similarity transformation on the matrix R ^*_Φ by

$$ {\mathbf{R}}_{\Upphi }^{*} = {\mathbf{P}}\Uplambda {\mathbf{P}}^{\text{T}} , $$

(33)

where Λ is a diagonal matrix made up of the eigenvalues of the matrix R ^*_Φ and P is an orthogonal matrix whose columns are the associated eigenvectors of R ^*_Φ . By the Wiener–Hopf filtering, we can define:

$$ {\mathbf{r}}_{d\Upphi }^{*} = {\mathbf{R}}_{\Upphi }^{*} {\mathbf{a}}_{\text{o}} , $$

(34)

where a _o is the optimum solution. Using the above equation for r ^*_dΦ substituting the orthogonal similarity transformation of (33) into (28), we get

$$ E\{ {\mathbf{a}}_{k} (n + 1)\} {\mathbf{P}}^{\text{T}} = \left[ {{\mathbf{I}} - \eta {\mathbf{P}}\Uplambda {\mathbf{P}}^{\text{T}} } \right]E\{ {\mathbf{a}}_{k} (n)\} {\mathbf{P}}^{\text{T}} + \eta \Uplambda {\mathbf{a}}_{\text{o}} {\mathbf{P}}^{\text{T}} . $$

(35)

Let w(n) be defined as a transformed version of the deviation between the $E\left\{ {{\mathbf{a}}_{k} (n)} \right\}$ and a _o . We may have the affine transformation as:

$$ E\{ {\mathbf{a}}_{k} (n)\} = {\mathbf{Pw}}(n) + {\mathbf{a}}_{\text{o}} , $$

(36)

and

$$ E\{ {\mathbf{a}}_{k} (n + 1)\} = {\mathbf{Pw}}(n + 1) + {\mathbf{a}}_{\text{o}} . $$

(37)

So, from (36), we simply to have

$$ \begin{aligned} ({\mathbf{Pw}}(n + 1) + {\mathbf{a}}_{\text{o}} ){\mathbf{P}}^{\text{T}} &= [{\mathbf{I}} - \eta {\mathbf{P}}\Uplambda {\mathbf{P}}^{\text{T}} ]({\mathbf{Pw}}(n) + {\mathbf{a}}_{\text{o}} ){\mathbf{P}}^{\text{T}} + \eta \Uplambda {\mathbf{a}}_{\text{o}} {\mathbf{P}}^{\text{T}} \\ {\mathbf{w}}(n + 1) + {\mathbf{a}}_{\text{o}} {\mathbf{P}}^{\text{T}} &= {\mathbf{w}}(n) - \eta \Uplambda {\mathbf{w}}_{k} (n) + {\mathbf{a}}_{\text{o}} {\mathbf{P}}^{\text{T}} - \eta \Uplambda {\mathbf{a}}_{\text{o}} {\mathbf{P}}^{\text{T}} + \eta \Uplambda {\mathbf{a}}_{\text{o}} {\mathbf{P}}^{\text{T}} \\ {\mathbf{w}}(n + 1) &= {\mathbf{w}}(n)[{\mathbf{I}} - \eta \Uplambda ] \\ \end{aligned}. $$

(38)

The above equation can represent a system of uncoupled homogeneous first-order difference equations as shown:

$$ w_{j} (n + 1) = w_{j} (n)\left( {1 - \eta \lambda_{j} } \right),\quad j = 1,2, \ldots ,(r + m), $$

(39)

where the λ _j are the eigenvalues of the matrix R ^*_Φ and w _j (n) is the jth element of the vector w(n). For the algorithm to be convergent in the mean, we require that for an arbitrary choice of the initial value of w _j (n) the following condition be satisfied:

$$ \left| {1 - \eta \lambda_{j} } \right| < 1.$$

(40)

Under this condition, if w _j (n) → 0 as n → ∞, so we can define the selection of the learning parameter η as follow

$$ 0 < \eta < \frac{1}{{\lambda_{\max } }}, $$

(41)

where λ _max is the largest eigenvalue of the matrix R ^*_Φ .

1.2 A.2. Proof of Condition 2 for PRNN convergence

The cost function defined in (21) can be expanded by a first order Taylor series as:

$$ J(n + 1) = J(n) + \left[ {\frac{\partial J}{{\partial {\mathbf{a}}_{k} }}} \right]\Updelta {\mathbf{a}}_{k} , $$

(42)

where $\Updelta {\mathbf{a}}_{k} = \eta \left[ { - \frac{\partial J}{{\partial {\mathbf{a}}_{k} }}} \right].$ By the convergence theorem of the second distinct, ΔJ ≤ 0 is given, so we define:

$$ \left[ {\frac{\partial J}{{\partial {\mathbf{a}}_{k} }}} \right]\Updelta {\mathbf{a}}_{k} \le 0. $$

(43)

Substituting (23) and (26) into (43), we get,

$$ \eta \left\| { - \left( {{\mathbf{d}}_{j} - {\mathbf{a}}_{k} (n)\Upphi_{j}^{R} } \right)\Upphi_{j}^{R} \frac{{\partial q^{ - l} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }} + \frac{2}{\beta }\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right|\exp \left( { - \frac{{\left\| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right\|^{2} }}{{\beta^{2} }}} \right)} \right\|^{2} \le 0, $$

(44)

and as η is supposed to be positive value, thus we have:

$$ \begin{gathered} - \left( {{\mathbf{d}}_{j} - {\mathbf{a}}_{k} (n)\Upphi_{j}^{R} } \right)\Upphi_{j}^{R} \frac{{\partial q^{ - l} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }} + \frac{2}{\beta }\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right|\exp \left( { - \frac{{\left\| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right\|^{2} }}{{\beta^{2} }}} \right) \le 0 \hfill \\ \Rightarrow \left( {{\mathbf{d}}_{j} - {\mathbf{a}}_{k} (n)\Upphi_{j}^{R} } \right)\Upphi_{j}^{R} \frac{{\partial q^{ - l} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }} \ge \frac{2}{\beta }\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right|\exp \left( { - \frac{{\left\| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right\|^{2} }}{{\beta^{2} }}} \right). \hfill \\ \end{gathered} $$

(45)

Taking the logarithm in both sides to get rid of exp, we have,

$$ \log_{\text {e}} \left\{ {\left( {{\mathbf{d}}_{j} - {\mathbf{a}}_{k} (n)\Upphi_{j}^{R} } \right)\Upphi_{j}^{R} \frac{{\partial q^{ - l} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }}} \right\} \ge \log_{e} \left\{ {\frac{2}{\beta }\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right|} \right\} - \frac{{\left\| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right\|^{2} }}{{\beta^{2} }}. $$

(46)

Initially, we assume that |β| ≫ 0 and is a quite large value, so β ² → ∞ then $\frac{1}{{\beta^{2} }} \to 0,$ therefore

$$ \left( {{\mathbf{d}}_{j} - {\mathbf{a}}_{k} (n)\Upphi_{j}^{R} } \right)\Upphi_{j}^{R} \frac{{\partial q^{ - l} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }} \ge \frac{2}{{\beta_{UB} }}\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right| $$

(47)

$$ \Rightarrow \quad \beta_{UB} \le 2\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right|\left[ {\left( {{\mathbf{d}}_{j} - {\mathbf{a}}_{k} (n)\Upphi_{j}^{R} } \right)\Upphi_{j}^{R} \frac{{\partial q^{ - l} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }}} \right]^{ - 1} . $$

(48)

So, we can choose the penalty factor β as follows

$$ 0 < \beta \le \left| {2\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right|\left[ {\left( {{\mathbf{d}}_{j} - {\mathbf{a}}_{k} (n)\Upphi_{j}^{R} } \right)\Upphi_{j}^{R} \frac{{\partial q^{ - l} {\mathbf{y}}_{k} }}{{\partial {\mathbf{a}}_{k} }}} \right]^{ - 1} } \right|. $$

(49)

Assume that it is working under a deep tree structure; Eq. (26) can substitute into (49) as:

$$ 0 < \beta \le \left| {2\left| {{\mathbf{a}}_{k} (n - 1) - {\mathbf{a}}_{k}^{*} } \right|\left[ {{\mathbf{e}}(n)\Upphi_{j}^{R} \Uplambda \left( {\Upphi_{j}^{1} } \right)\Upphi_{j}^{1T} {\mathbf{Q}}} \right]^{ - 1} } \right|. $$

(50)