FF-SKPCCA: Kernel probabilistic canonical correlation analysis

Abstract

Several information fusion methods are developed for increasing the recognition accuracy in multimodal systems. Canonical correlation analysis (CCA), cross-modal factor analysis (CFA) and their kernel versions are known as successful fusion techniques but they cannot digest the data variability. Probabilistic CCA (PCCA) is suggested as a linear fusion method to capture input variability. A new kernel PCCA (KPCCA) is proposed here to capture both the nonlinear correlations of sources and input variability. The functionality of KPCCA decreases when the number of samples, which determines the size of kernel matrix increases. In the conventional fusion methods the latent variables of different modalities are concatenated; consequently, a large-scale covariance matrix with just limited number of samples must be estimated To overcome this drawback, a sparse KPCCA (SKPCCA) is introduced which scarifies the covariance matrix elements at the cost of decreasing its rank. In the final stage of the gradual evolution of KPCCA, a new feature fusion manner is proposed for SKPCCA (FF-SKPCCA) as a second stage fusion. This proposed method unifies the latent variables of two modalities into a feature vector with an acceptable size. Audio-visual databases like M2VTS (for speech recognition) eNTERFACE and RML (for emotion recognition) are applied to assess FF-SKPCCA compared to state-of-the-art fusion methods. The comparative results indicate the superiority of the proposed method in most cases.

Appendix A

1.1 A-1. CCA Method

A proposed statistical method named Canonical Correlation Analysis (CCA) is proposed by [14], in order to find a shared structure between two sources of data. CCA is closely related to the mutual information method [45] but it has some differences in terms of objective function. A pair of feature vectors with zero is considered in the method as follows:

$$\begin{array}{@{}rcl@{}} \left( \mathbf{x},\mathbf{y} \right)=\left\{ \left( x_{1},y_{1} \right),\left( x_{2},y_{2} \right),\mathellipsis ,\left( x_{n},y_{n} \right) \right\} \end{array} $$

(15)

where x _i and y _i are the observation data (original features) of the two modalities with dimensions of p and q, respectively. CCA seeks to develop two transformation matrices of W _x and W _y with dimensions of p×d and q×d respectively, where d≤min(p q). The original features of these modalities are projected to the correlation subspace by W _x and W _y in a manner that the correlation between $\hat {x}=\mathbf {x}\mathbf {W}_{x}$ and $\hat {y}=\mathbf {y}\mathbf {W}_{y}$ is maximized. Maximizing the correlation between the projected feature vectors of $\hat {x}$ and $\hat {y}$ is the same as maximizing ρ(correlation coefficient) between them as follows:

$$\begin{array}{@{}rcl@{}} \mathrm{\rho }\!&=&\!\max\limits_{\mathbf{W}_{x},\mathbf{W}_{y}}\frac{E\left[ \hat{x}^{\mathbf{T}}\hat{y} \right]}{\sqrt {E\left[ \hat{x}^{\mathbf{2}} \right]E\left[ \hat{y}^{\mathbf{2}} \right]} } \end{array} $$

$$\begin{array}{@{}rcl@{}} \!&=&\!\max\limits_{\mathbf{W}_{x},\mathbf{W}_{y}} \frac{E\left[ \mathbf{W}_{\mathbf{x}}^{\mathbf{T}}\mathbf{x}^{\mathbf{T}}\mathbf{y}\mathbf{W}_{\mathbf{y}} \right]}{\sqrt {E\left[ \mathbf{W}_{\mathbf{x}}^{\mathbf{T}}\mathbf{x}^{\mathbf{T}}\mathbf{x}\mathbf{W}_{\mathbf{x}} \right]E\left[ \mathbf{W}_{\mathbf{y}}^{\mathbf{T}}\mathbf{y}^{\mathbf{T}}\mathbf{y}\mathbf{W}_{\mathbf{y}} \right]}} \end{array} $$

$$\begin{array}{@{}rcl@{}} &=&\max\limits_{\mathbf{W}_{x},\mathbf{W}_{y}}\frac{\mathbf{W}_{x}^{T}C_{xy}\mathbf{W}_{y}}{\sqrt {\mathbf{W}_{x}^{T}C_{xx}\mathbf{W}_{x}\mathbf{W}_{y}^{T}C_{yy}\mathbf{W}_{y}} } \end{array} $$

(16)

where C _{x y} is the cross-covariance matrix of (x , y) and C _{x x} , C _{y y} are the covariance matrices of x and y respectively.

The above equation can be solved as an Eigen-value problem like:

$$\begin{array}{@{}rcl@{}} C_{xx}^{\mathrm{-1}}C_{xy}C_{yy}^{\mathrm{-1}}C_{yx}\mathbf{W}_{x}\mathrm{=}\rho^{\mathrm{2}}\mathbf{W}_{x} \end{array} $$

(17)

$$\begin{array}{@{}rcl@{}} C_{yy}^{-1}C_{yx}C_{xx}^{-1}C_{xy}\mathbf{W}_{y}= \rho^{2}\mathbf{W}_{y} \end{array} $$

(18)

1.2 A-2. CFA Method

The cross-modal factor analysis (CFA) method is proposed by [15], where, the features from different modalities are treated as two subsets where the same patterns between these two subsets are discovered. In this method, it is assumed that a pair of normalized feature vectors x and y with zero means are linearly projected into a joint space applying W _x and W _y transforms, in a manner that the following criterion can be minimizing:

$$\begin{array}{@{}rcl@{}} {\underset{{W}_{x},{W}_{y}}{\min}}\left\| \mathbf{x}\mathbf{W}_{x}-\mathbf{y}\mathbf{W}_{y} \right\|_{F}^{2} \end{array} $$

(19)

where, $\mathbf {W}_{x}^{T}\mathbf {W}_{x}$ and $\mathbf {W}_{y}^{T}\mathbf {W}_{y}$ are unit matrices and F is the Frobenius norm and is calculated by $\left \| \mathbf {W} \right \|_{F}=\sqrt {\sum \nolimits _{ij} w_{ij}^{2}}$.

By solving the above equation for optimal transformation matrices W _x and W _y and decomposing cross-covariance matrix C _{x y} through Singular Value Decomposition (SVD) method, the following equation is obtained:

$$\begin{array}{@{}rcl@{}} C_{xy}=S_{xy}{\Lambda}_{xy}D_{xy} \end{array} $$

(20)

Consequently,

$$\begin{array}{@{}rcl@{}} \mathbf{W}_{x}= S_{xy}\& \mathbf{W}_{y}=D_{xy} \end{array} $$

(21)

1.3 A-3. Probabilistic CCA

To deal with the uncertainty problem in the CCA performance, the probabilistic CCA (PCCA) is introduced by [24] through the projected latent variables provide maximum variance in the joint correlation space. To do this, they defined a Gaussian model for every single source of data as follow:

$$\begin{array}{@{}rcl@{}} z&\sim& \mathcal{N}\left( {\mathrm{0,}I}_{d} \right)\mathrm{ 1\le }d\mathrm{\le }\min \left( p\mathrm{,}q \right) \end{array} $$

$$\begin{array}{@{}rcl@{}} \mathbf{x\vert }z&=&\mathcal{N}\left( {z\mathbf{W}}_{x}^{T} +\mu_{x},\varphi_{x} \right) \end{array} $$

$$\begin{array}{@{}rcl@{}} \mathbf{y\vert }z&=&\mathcal{N}({z\mathbf{W}}_{y}^{T} +\mu_{y}\varphi_{y}) \end{array} $$

(22)

where, z is the latent variable, shared between the two modalities x and y and μ and φ are the mean and covariance of each data, respectively. Here, by maximizing the probability functions, the φ _x and φ _y should be minimized. By considering $\mathbf {O=}\left [ {\begin {array}{*{20}c} \mathbf {x}\\ \mathbf {y}\\ \end {array} } \right ]$ , W = [ W _x W _y ] , $\mu \mathbf {=}\left [ {\begin {array}{*{20}c} \mu _{x}\\ \mu _{y}\\ \end {array} } \right ]\textit {and}\varphi \mathbf {=}\left [ {\begin {array}{*{20}c} \varphi _{x} & \mathbf {0}\\ \mathbf {0} & \varphi _{y}\\ \end {array} } \right ]$both probabilistic functions are merged into the following joint probabilistic function as:

$$\begin{array}{@{}rcl@{}} \mathbf{p}\left( \mathbf{O,z} \right)=\mathbf{p}(\mathbf{O}\vert \mathbf{z})\mathbf{p}(z)= N\left( z\mathbf{W}^{T} +\mu ,\varphi \right)\mathrm{N}\left( {\mathrm{0,}I}_{d} \right) \end{array} $$

(23)

They indicate that the posterior expectation of z given x and y are:

$$\begin{array}{@{}rcl@{}} {\varphi }_{x}&=&C_{xx}-\mathbf{W}_{x}\mathbf{W}_{x}^{T} \end{array} $$

$$\begin{array}{@{}rcl@{}} {\varphi }_{y}&=&C_{yy}-\mathbf{W}_{y}\mathbf{W}_{y}^{T} \end{array} $$

$$\begin{array}{@{}rcl@{}} E\left( z \vert \mathbf{x}\right) &=& \mathbf{x}\mathbf{W}_{x}M_{x}^{-1}, M_{x}=I+ \mathbf{W}_{x}^{T}\varphi_{x}^{-1}\mathbf{W}_{x} \end{array} $$

$$\begin{array}{@{}rcl@{}} E\left( z \vert \mathbf{y}\right) &=& \mathbf{y}\mathbf{W}_{y}M_{y}^{-1} , M_{y}=I+ \mathbf{W}_{y}^{T}\varphi_{y}^{-1}\mathbf{W}_{y} \end{array} $$

(24)

where, W _x and W _y are the first d canonical directions of x and y. The parameters C _{x x} and C _{y y} are the covariances of x and y, respectively.

However, this new method named the unified latent variable can identify a latent variable, given x and y as:

$$\begin{array}{@{}rcl@{}} E\left( z \vert {\mathbf{x,y}}\right) \!\!&=&\!\!\left[ {\begin{array}{*{20}c} \mathbf{x}\mathbf{W}_{x} & \mathbf{y}\mathbf{W}_{y}\\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {(I\,-\,{P_{d}^{2}})}^{-1} & {(I\,-\,{P_{d}^{2}})}^{-1}P_{d}\\ {(I\,-\,{P_{d}^{2}})}^{-1}P_{d} & {(I\,-\,{P_{d}^{2}})}^{-1}\\ \end{array} } \right]\\ &&\times\left[ {\begin{array}{*{20}c} M_{x}^{-1}\\M_{y}^{-1}\\ \end{array} } \right] \end{array} $$

(25)

where, $P_{d}=M_{x}^{-1}\ast {(M_{y}^{-1})}^{T}$.

Another solution for (22) is based on the expectation maximization (EM) algorithm. Similarly, the Probabilistic Principle Component Analysis (PPCA) is proposed by [46] which iterates through the following expectation maximization (EM).

• Expectation-step: finds the sufficient statistics of the latent variables given the current estimated parameter:

  $$\begin{array}{@{}rcl@{}} M_{t}&=&I+ \mathbf{W}_{t}^{T}\varphi_{t}^{-1}\mathbf{W}_{t} \end{array} $$
  $$\begin{array}{@{}rcl@{}} E(z_{t})&=& M_{t}^{-1}\mathbf{W}_{t}\varphi_{t}^{-1}\mathbf{O} \end{array} $$
  $$\begin{array}{@{}rcl@{}} E(z_{t}{z_{t}^{T}})&=&M_{t}^{-1}+E(z_{t}){E(z_{t})}^{T} \end{array} $$

  (26)

  where subscript t indicate the iteration number.

• Maximization-step: updates the estimated parameter to maximize the likelihood function:

$$\begin{array}{@{}rcl@{}} \mathbf{W}_{\mathbf{t+1}}&=&\left[ \mathbf{O}{E(z_{t})}^{T} \right]\left[ E(z_{t}{z_{t}^{T}}) \right]^{-1} \end{array} $$

$$\begin{array}{@{}rcl@{}} \mathrm{\varphi }_{t+1}&=&\mathbf{O}\mathbf{O}^{\mathbf{T}}-2\mathbf{O}{E(z_{t})}^{T}\mathbf{W}_{t+1}^{T}\\ &&+ ~\mathbf{trace}(E(z_{t}{z_{t}^{T}}){\mathbf{W}_{t+1}^{T}\mathbf{W}}_{\mathbf{t+1}}\mathbf{)} \end{array} $$

(27)

By inserting (26) into (27), this method provides a general solution for PCCA scheme which yields the following updated equation:

$$\begin{array}{@{}rcl@{}} \mathbf{W}_{\mathbf{t}\mathrm{\mathbf{+1}}}\!\!&=&\!\!C\mathrm{\varphi }_{t}^{\mathrm{-1}}\mathbf{W}_{\mathbf{t}}M_{t}^{\mathrm{-1}}\left( M_{t}^{\mathrm{-1}}\mathrm{\!+}M_{t}^{\mathrm{-1}}\mathbf{W}_{t}^{T}\mathrm{\varphi }_{t}^{\mathrm{-1}}C\mathrm{\varphi }_{t}^{\mathrm{-1}}\mathbf{W}_{\mathbf{t}}M_{t}^{\mathrm{-1}} \right)^{\mathrm{-1}} \end{array} $$

$$\begin{array}{@{}rcl@{}} \mathrm{\varphi }_{t+1}\!\!&=&\!\! \left( {\begin{array}{*{20}c} \left( C\,-\,C\mathrm{\varphi }_{t}^{-1}\mathbf{W}_{\mathbf{t}}M_{t}^{-1}\mathbf{W}_{t+1}^{T} \right)_{11} & 0\\ 0 & \left( C\,-\,C\mathrm{\varphi }_{t}^{-1}\mathbf{W}_{\mathbf{t}}M_{t}^{-1}\mathbf{W}_{t+1}^{T} \right)_{22}\\ \end{array} } \right)\\ \end{array} $$

(28)

where $M_{t}\mathrm {=}I\mathrm {+ }\mathbf {W}_{t}^{T}\mathrm {\varphi }_{t}^{\mathrm {-1}}\mathbf {W}_{t}$ .

1.4 A-4. KCCA Method

Kernel Canonical Correlation Analysis (KCCA) [21] is the kernelized version of CCA method that projects data into higher dimensional feature spaces and applies CCA to the data in the kernel space in order to find a nonlinear correlation between the two modalities. Let us consider ϕ and ψ as two mapping functions that map the input data into a space of higher dimension:

$$\begin{array}{@{}rcl@{}} \left( \mathrm{\phi \mathbf{(x)}},\psi (\mathrm{\mathbf{y)}} \right)&=&\left\{ \left( \mathrm{\phi (}\mathrm{x}_{1}),\psi (\mathrm{y}_{1}) \right),\left( \mathrm{\phi (}\mathrm{x}_{2}),\psi (\mathrm{y}_{2}) \right),\right.\\ &&\left.\mathellipsis ,\left( \mathrm{\phi (}\mathrm{x}_{n}),\psi (\mathrm{y}_{n}) \right) \right\} \end{array} $$

(29)

The KCCA seeks to develop the two matrices α and β that are applied in the following equations:

$$\begin{array}{@{}rcl@{}} \mathbf{W}_{x}={\mathrm{\phi }(\mathrm{\mathbf{x}})}^{T}\mathbf{\alpha } \end{array} $$

(30)

$$\begin{array}{@{}rcl@{}} \mathbf{W}_{y}={\mathrm{\psi }(\mathrm{\mathbf{y}})}^{T}\mathbf{\beta } \end{array} $$

(31)

This means that W _x and W _y are the projections of ϕ(x) and ψ(y) onto α and β, respectively. By inserting ϕ and ψ, into (16), the correlation function is applied as:

$$\begin{array}{@{}rcl@{}} \mathrm{\rho } \!\!&=&\!\! {\underset{\boldsymbol{\alpha,\beta}}{\max}}\frac{E\left[ \mathbf{\alpha }^{T}\mathrm{\phi }\left( \mathrm{x} \right){\mathrm{.\phi }\left( \mathrm{x} \right)}^{T}\mathrm{\psi }\left( \mathrm{y} \right){\mathrm{.\psi }\left( \mathrm{y} \right)}^{T}\mathbf{\beta } \right]}{\sqrt {E\left[ \mathbf{\alpha }^{T}\mathrm{\phi }\left( \mathrm{x} \right){\mathrm{.\phi }\left( \mathrm{x} \right)}^{T}\mathrm{\phi }\left( \mathrm{x} \right){\mathrm{.\phi }\left( \mathrm{x} \right)}^{T}\mathbf{\alpha } \right]E\left[ \mathbf{\beta }^{T}\mathrm{\psi }\left( \mathrm{y} \right){\mathrm{.\psi }\left( \mathrm{y} \right)}^{T}\mathrm{\psi }\left( \mathrm{y} \right){\mathrm{.\psi }\left( \mathrm{y} \right)}^{T}\mathbf{\beta } \right]} } \end{array} $$

$$\begin{array}{@{}rcl@{}} \!\!&=&\!\!{\underset{\alpha,\beta}{\max}}\frac{\mathbf{\alpha }^{T}\mathbf{K}_{\mathrm{x}}\mathbf{K}_{\mathrm{y}}\mathbf{\beta }}{\sqrt {\left[ \mathbf{\alpha }^{T}\mathbf{K}_{\mathrm{x}}\mathbf{K}_{\mathrm{x}}\mathbf{\alpha } \right]\left[ \mathbf{\beta }^{T}\mathbf{K}_{\mathrm{y}}\mathbf{K}_{\mathrm{y}}\mathbf{\beta } \right]} } \end{array} $$

(32)

where, K _x = E[ϕ(x).ϕ(x)^T] and K _y = E[ψ(y).ψ(y)^T].

This optimization problem can be solved through the generalized Eigen-value decomposition method. When kernel functions are non-invertible, conventional regularization technique can be applied, therefore, the following Equation is yield [30]:

$$\begin{array}{@{}rcl@{}} {\underset{\alpha,\beta}{\max}}\frac{\boldsymbol{\alpha }^{T}\mathbf{K}_{\mathrm{x}}\mathbf{K}_{\mathrm{y}}\boldsymbol{\beta }}{\sqrt {\left[ \boldsymbol{\alpha }^{T}\mathrm{(}\mathbf{K}_{\mathrm{x}}^{\mathrm{2}}\mathrm{ +\tau }\mathbf{K}_{\mathrm{x}}\mathrm{)}\boldsymbol{\alpha } \right]\left[ \boldsymbol{\beta }^{T}\mathrm{(}\mathbf{K}_{\mathrm{y}}^{\mathrm{2}}\mathrm{ +\tau }\mathbf{K}_{\mathrm{y}}\mathrm{)}\boldsymbol{\beta } \right]} } \end{array} $$

(33)

where, 0≤τ≤1.

1.5 A-5. KCFA Method

Kernel CFA [12] approach can provide correct information association provided that the two modalities are not linearly related. To illustrate this fact X=(ϕ(x ₁), ϕ(x ₂),…,ϕ(x _n ))^T, and Y =(ψ(y ₁),ψ(y ₂),…,ψ(y _n ))^T represent the two matrices with each row representing a sample in the nonlinearly mapped feature space; next, the X ^TY = S _{x y} ##Λ## _{x y} D _{x y} should be solved through kernel method. The kernel matrices of the two subsets of features can be computed as K _x = X X ^T and K _y = Y Y ^T. By performing eigenvalue decomposition on the product of the kernel matrices K _x K _y , it becomes obvious that

$$\begin{array}{@{}rcl@{}} (K_{x}K_{y}) \boldsymbol{\beta }_{i}&=& \mathrm{\lambda }_{i}\boldsymbol{\beta }_{i} \end{array} $$

$$\begin{array}{@{}rcl@{}} \left( X X^{T}Y Y^{T} \right)\boldsymbol{\beta }_{i}&=&\mathrm{\lambda }_{i}\boldsymbol{\beta }_{i} \end{array} $$

$$\begin{array}{@{}rcl@{}} \left( Y^{T}X X^{T}Y \right)Y^{T}\boldsymbol{\beta }_{i}&=&\mathrm{\lambda }_{i}Y^{T}\boldsymbol{\beta }_{i} \end{array} $$

(34)

Since the right singular vectors of the SVD of X ^T Y,D _{x y} correspond with the eigenvectors of Y ^T X X ^T Y= (X ^T Y)^T(X ^T Y)Y ^T β _i corresponds to the columns of D _{x y} , which can be further normalized into a unit norm as:

$$\begin{array}{@{}rcl@{}} v_{i}=\frac{Y^{T}\boldsymbol{\beta }_{i}}{\left\| Y^{T}\boldsymbol{\beta }_{i} \right\|}= \frac{Y^{T}\boldsymbol{\beta }_{i}}{\sqrt {\boldsymbol{\beta }_{i}^{T}YY^{T}\boldsymbol{\beta }_{i}} }= \frac{Y^{T}\boldsymbol{\beta }_{i}}{\sqrt {\boldsymbol{\beta }_{i}^{T}K_{y}\boldsymbol{\beta }_{i}} } \end{array} $$

(35)

For a feature vector y ^′ with nonlinear mapping ψ(y ^′), the projection can be computed as

$$\begin{array}{@{}rcl@{}} {v_{i}^{T}}\mathrm{\psi }\left( y^{\prime} \right)&=&\left( \frac{Y^{T}\boldsymbol{\beta }_{i}}{\sqrt {\boldsymbol{\beta }_{i}^{T}K_{y}\boldsymbol{\beta }_{i}} } \right)^{T}\mathrm{\psi }\left( y^{\prime} \right) \end{array} $$

$$\begin{array}{@{}rcl@{}} &=& \frac{1}{\sqrt {\boldsymbol{\beta }_{i}^{T}K_{y}\boldsymbol{\beta }_{i}} }\boldsymbol{\beta }_{i}^{T}\left[ {\begin{array}{l} K\left( y^{\prime},y_{1} \right) \\ K\left( y^{\prime},y_{2} \right) \\ \mathellipsis \\ K(y^{\prime},y_{n}) \\ \end{array}} \right] \end{array} $$

(36)

Similarly, it can be illustrated that

$$\begin{array}{@{}rcl@{}} (K_{y}K_{x})\mathbf{\alpha }_{j}=\left( X^{T}Y Y^{T}X \right)X^{T}\mathbf{\alpha }_{j}= \mathrm{\lambda }_{j}X^{T}\mathbf{\alpha }_{j} \end{array} $$

(37)

The left singular vectors S _{x y} are the eigenvectors of X ^T Y Y ^T X=(X ^T Y)(X ^T Y)^T, hence X ^T α _j corresponds to the S _{x y} columns, which can be normalized into a unit norm as:

$$\begin{array}{@{}rcl@{}} \boldsymbol{\mu }_{j}=\frac{X^{T}\mathbf{\alpha }_{j}}{\left\| X^{T}\mathbf{\alpha }_{j} \right\|}= \frac{X^{T}\mathbf{\alpha }_{j}}{\sqrt {\mathbf{\alpha }_{i}^{T}XX^{T}\mathbf{\alpha }_{j}} }= \frac{X^{T}\mathbf{\alpha }_{j}}{\sqrt {\mathbf{\alpha }_{i}^{T}K_{x}\mathbf{\alpha }_{j}} } \end{array} $$

(38)

By allowing x ^′ to be a feature vector in the original domain where the nonlinear mapping is ϕ(x ^′), the feature vector in the cross-modal associated domain can be computed as:

$$\begin{array}{@{}rcl@{}} \boldsymbol{\mu }_{j}^{T}\mathrm{\phi (}x^{\prime}\mathrm{)}&=& \left( \frac{X^{T}\mathbf{\alpha }_{j}}{\sqrt {\mathbf{\alpha }_{j}^{T}K_{x}\mathbf{\alpha }_{j}} } \right)^{T}\mathrm{ \phi (}x^{\prime}\mathrm{)} \end{array} $$

$$\begin{array}{@{}rcl@{}} &=& \frac{1}{\sqrt {\mathbf{\alpha }_{j}^{T}K_{x}\mathbf{\alpha }_{j}} }\boldsymbol{\alpha }_{j}^{T}\left[ {\begin{array}{l} K\left( x^{\prime},x_{1} \right) \\ K\left( x^{\prime},x_{2} \right) \\ \mathellipsis \\ K(x^{\prime},x_{n}) \\ \end{array}} \right] \end{array} $$

(39)