Dual structural consistency based multi-modal correlation propagation projections for data representation

Abstract

Canonical correlation analysis (CCA) is a powerful tool for analyzing multi-dimensional paired data. However, when facing semi-supervised multi-modal data (Also called multi-view Hou et al. (Pattern Recog 43(3):720–730, 2010) or multi-represented Kailing et al. (Clustering multi-represented objects with noise. In: Proceedings of the eighth Pacific-Asia conference on knowledge discovery and data mining (PAKDD). Sydney, Australia, pp 394–403) data. For convenience, we will uniformly call them multi-modal data hereafter.) which widely exist in real-world applications, CCA usually performs poorly due to ignoring useful supervised information. Meanwhile, due to the limited labeled training samples in the semi-supervised scenario, supervised extensions of CCA suffer from overfitting. Several semi-supervised extensions of CCA have been proposed recently. Nevertheless, they either just utilize the global structural information captured from the unlabeled data, or propagate label information by discovering the affinities just between the labeled and unlabeled data points in advance. In this paper, we propose a robust multi-modal semi-supervised feature extraction and fusion framework, termed as dual structural consistency based multi-modal correlation propagation projections (SCMCPP). SCMCPP guarantees the consistency between representation structure and hypotaxis structure in each modality and ensures the consistency of hypotaxis structure between two different modalities. By iteratively propagating labels and learning affinities, discriminative information of both given labels and estimated labels is utilized to improve the affinity construction and infer the remaining unknown labels. Moreover, probabilistic within-class scatter matrices in each modality and probabilistic correlation matrix between two modalities are constructed to enhance the discriminative power of features. Extensive experiments on several benchmark face databases demonstrate the effectiveness of our approach.

Appendix

The proof of Theorem 1

We can derive the formula from Eqs. (6) and (7) as

$$ {\left\Vert \Gamma \odot A\right\Vert}_1={\displaystyle {\sum}_{i,j}\left|{\Gamma}_{ij}\cdot {A}_{ij}\right|={\displaystyle {\sum}_{i,j}\left|{\Gamma}_{ij}\cdot \frac{1}{2}\left(\left|{W}_{ij}\right|+\left|{W}_{ji}\right|\right)\right|}}. $$

Owing to ${\Gamma}_{ij}=\frac{1}{2}{\left\Vert {p}_i-{p}_j\right\Vert}^2={\Gamma}_{ji}\ge 0$, we can get that

$$ \begin{array}{ll}{\left\Vert \varGamma \odot A\right\Vert}_1\hfill & \begin{array}{l}=\kern0.5em {\displaystyle {\sum}_{i,j}\frac{1}{2}\left(\left|{\varGamma}_{ij}\cdot \left|{W}_{ij}\right|\right|+\left|{\varGamma}_{ji}\cdot \left|{W}_{ji}\right|\right|\right)}\hfill \\ {}={\displaystyle {\sum}_{i,j}\left|{\varGamma}_{ij}\cdot \left|{W}_{ij}\right|\right|}\hfill \\ {}={\displaystyle {\sum}_{i,j}\left|{\varGamma}_{ij}\cdot {W}_{ij}\right|}\hfill \\ {}={\left\Vert \varGamma \odot W\right\Vert}_1\hfill \end{array}\hfill \end{array} $$

The proof of Theorem 2

Owing to the fact that ${\overset{\sim }{S}}_w^{(X)}$ and ${\overset{\sim }{S}}_w^{(Y)}$ are both non-singular square matrices, we can get that $rank\left({\left({\overset{\sim }{S}}_w^{(X)}\right)}^{-1}{\overset{\sim }{L}}_{xy}{\left({\overset{\sim }{S}}_w^{(Y)}\right)}^{-1}{\overset{\sim }{L}}_{xy}^T\right)= rank\left({\left({\overset{\sim }{S}}_w^{(Y)}\right)}^{-1}{\overset{\sim }{L}}_{xy}^T{\left({\overset{\sim }{S}}_w^{(X)}\right)}^{-1}{\overset{\sim }{L}}_{xy}\right)= rank\left({\overset{\sim }{L}}_{xy}\right)$. And we obtain the following inequality

$$ rank\left({\tilde{L}}_{xy}\right)\le min\left\{ rank\left(X{P}^{(X)T}\right), rank\left({P}^{(Y)}{Y}^T\right)\right\}. $$

Furthermore, let ${P}_{i\circ}^{(X)}$ denote the i-th row of ${P}^{(X)}$, where $i=1,2,\cdots, c$. Then we can obtain the following equality:

$$ X{P}^{(X)T}=X\cdot \left[{P}_{1\circ}^{(X)T},{P}_{2\circ}^{(X)T},\cdots, {P}_{c\circ}^{(X)T}\right]=\left[X{P}_{1\circ}^{(X)T},X{P}_{2\circ}^{(X)T},\cdots, X{P}_{c\circ}^{(X)T}\right]. $$

From the definition of the hard label and Eq. (16), for each ${p}_j^{(X)}$ in ${P}^{(X)}=\left[{p}_1^{(X)},{p}_2^{(X)},\cdots, {p}_N^{(X)}\right]$ where $j=1,2,\cdots, N$, we have ${\sum}_{i=1}^c{p}_{j,i}^{(X)}=1$. So we can obtain the following equality:

$$ {P}_{1\circ}^{(X)T}+{P}_{2\circ}^{(X)T}+\cdots +{P}_{c\circ}^{(X)T}=\left[{\displaystyle {\sum}_{i=1}^c{p}_{1,i}^{(X)};}{\displaystyle {\sum}_{i=1}^c{p}_{2,i}^{(X)}};\cdots; {\displaystyle {\sum}_{i=1}^c{p}_{N,i}^{(X)}}\right]={1}_{N\times 1} $$

Then

$$ X{P}_{1\circ}^{(X)T}+X{P}_{2\circ}^{(X)T}+\cdots +X{P}_{c\circ}^{(X)T}=X\cdot {1}_{N\times 1}={\displaystyle {\sum}_{j=1}^N{x}_j={0}_{p\times 1}} $$

which means that all $X{P}_{i\circ}^{(X)T}$ are linearly dependent. Therefore, $rank\left(X{P}^{(X)T}\right)\le min\left\{v,c-1\right\}$. Similarly, we can get $rank\left({P}^{(Y)}{Y}^T\right)\le min\left\{q,c-1\right\}$. Thus, we have $rank\left({\overset{\sim }{L}}_{xy}\right)\le min\left\{v,q,c-1\right\}$

Owing to the fact that c is always much less than v and q, we have $rank\left({\overset{\sim }{L}}_{xy}\right)\le c-1$. From Eq. (26), α and β are calculated as the eigenvectors of ${\left({\overset{\sim }{S}}_w^{(X)}\right)}^{-1}{\overset{\sim }{L}}_{xy}{\left({\overset{\sim }{S}}_w^{(Y)}\right)}^{-1}{\overset{\sim }{L}}_{xy}^T$ and ${\left({\overset{\sim }{S}}_w^{(Y)}\right)}^{-1}{\overset{\sim }{L}}_{xy}^T{\left({\overset{\sim }{S}}_w^{(X)}\right)}^{-1}{\overset{\sim }{L}}_{xy}$, respectively. Therefore, we infer that in SCMCPP, there are at most $c-1$ pairs of projection directions.