Multi-nonlinear multi-view locality-preserving projection with similarity learning for random cross-view gait recognition

Abstract

View variation is one of the greatest challenges in the field of gait recognition. Subspace learning approaches are designed to solve this issue by projecting cross-view features into a common subspace before recognition. However, similarity measures are data-dependent, which results in low accuracy when cross-view gait samples are randomly arranged. Inspired by the recent developments of data-driven similarity learning and multi-nonlinear projection, we propose a new unsupervised projection approach, called multi-nonlinear multi-view locality-preserving projections with similarity learning (M²LPP-SL). The similarity information among cross-view samples can be learned adaptively in our M²LPP-SL. Besides, the complex nonlinear structure of original data can be well preserved through multiple explicit nonlinear projection functions. Nevertheless, its performance is largely affected by the choice of nonlinear projection functions. Considering the excellent ability of kernel trick for capturing nonlinear structure information, we further extend M²LPP-SL into kernel space, and propose its multiple kernel version MKMLPP-SL. As a result, our approaches can capture linear and nonlinear structure more precisely, and also learn similarity information hidden in the multi-view gait dataset. The proposed models can be solved efficiently by alternating direction optimization method. Extensive experimental results over various view combinations on the multi-view gait database CASIA-B have demonstrated the superiority of the proposed algorithms.

Appendix

Derivation of transforming the objective function of the proposed M²LPP-SL into problem (12) when S^XY is updated and fixed:

$$\begin{aligned} J(P_{\text{X}} ,P_{\text{Y}} ) & = \frac{1}{2}\sum\limits_{i,p} {\left\| {P_{\text{X}}^{\text{T}} \varPhi (x_{i} ) - P_{\text{X}}^{\text{T}} \varPhi (x_{p} )} \right\|}^{2} S_{ip}^{\text{X}} + \frac{1}{2}\sum\limits_{j,q} {\left\| {P_{\text{Y}}^{\text{T}} \varPhi (y_{j} ) - P_{\text{Y}}^{\text{T}} \varPhi (y_{q} )} \right\|}^{2} S_{jq}^{\text{Y}} \\ &\quad \;\,{ + }\sum\limits_{i,j} {\left\| {P_{\text{X}}^{\text{T}} \varPhi (x_{i} ) - P_{\text{Y}}^{\text{T}} \varPhi (y_{j} )} \right\|}^{2} S_{ij}^{\text{XY}} \;\;\;\;\;\;\;\;\; \\ & = \;\frac{1}{2} \times 2{\text{Tr}}\left[ {P_{\text{X}}^{\text{T}} \varPhi (X)(D^{\text{X}} - S^{\text{X}} )\varPhi (X)^{\text{T}} P_{\text{X}} } \right]{ + }\frac{1}{2} \times 2{\text{Tr}}\left[ {P_{\text{Y}}^{\text{T}} \varPhi (Y)(D^{\text{Y}} - S^{\text{Y}} )\varPhi (Y)^{\text{T}} P_{\text{Y}} } \right] \\ & \quad\;\;{ + }\;{\text{Tr}}\left[ {P_{\text{X}}^{\text{T}} \varPhi (X)D^{{ ( {\text{XY)}}_{\text{r}} }} \varPhi (X)^{\text{T}} P_{\text{X}} { + }P_{\text{Y}}^{\text{T}} \varPhi (Y)D^{{ ( {\text{XY)}}_{\text{c}} }} \varPhi (Y)^{\text{T}} P_{\text{Y}} } \right. \\ & \quad\;\;\left. { - P_{\text{X}}^{\text{T}} \varPhi (X)S^{\text{XY}} \varPhi (Y)^{\text{T}} P_{\text{Y}} \; - P_{\text{Y}}^{\text{T}} \varPhi (Y)(S^{\text{XY}} )^{\text{T}} \varPhi (X)^{\text{T}} P_{\text{X}} } \right] \\ & { = }\;{\text{Tr}}\left[ {P_{\text{X}}^{\text{T}} \varPhi (X)\left( {D^{{ ( {\text{XY)}}_{\text{r}} }} + D^{\text{X}} - S^{\text{X}} } \right)\varPhi (X)^{\text{T}} P_{\text{X}} + P_{\text{Y}}^{\text{T}} \varPhi (Y)\left( {D^{{ ( {\text{XY)}}_{\text{c}} }} + D^{\text{Y}} - S^{\text{Y}} } \right)\varPhi (Y)^{\text{T}} P_{\text{Y}} } \right. \\ &\quad \;\;\left. { - P_{\text{X}}^{\text{T}} \varPhi (X)S^{\text{XY}} \varPhi (Y)^{\text{T}} P_{\text{Y}} - P_{\text{Y}}^{\text{T}} \varPhi (Y)(S^{\text{XY}} )^{\text{T}} \varPhi (X)^{\text{T}} P_{\text{X}} } \right] \\ & = \;{\text{Tr}}\left\{ {\left[ {\begin{array}{*{20}c} {P_{\text{X}}^{\text{T}} \varPhi (X)} & {P_{\text{Y}}^{\text{T}} \varPhi (Y)} \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {D^{{ ( {\text{XY)}}_{\text{r}} }} + D^{\text{X}} } & {} \\ {} & {D^{{ ( {\text{XY)}}_{\text{c}} }} + D^{\text{Y}} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {\varPhi (X)^{\text{T}} P_{\text{X}} } \\ {\varPhi (Y)^{\text{T}} P_{\text{Y}} } \\ \end{array} } \right]} \right\} \\ & \;\; - {\text{Tr}}\left\{ {\left[ {\begin{array}{*{20}c} {P_{\text{X}}^{\text{T}} \varPhi (X)} & {P_{\text{Y}}^{\text{T}} \varPhi (Y)} \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {S^{\text{X}} } & {S^{\text{XY}} } \\ {(S^{\text{XY}} )^{\text{T}} } & {S^{\text{Y}} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {\varPhi (X)^{\text{T}} P_{\text{X}} } \\ {\varPhi (Y)^{\text{T}} P_{\text{Y}} } \\ \end{array} } \right]} \right\}, \\ \end{aligned}$$

(20)

where S^X = [$s_{ip}^{\text{X}}$]_n × n is the similarity matrix within X-view, the degree matrix with $d_{ii}^{\text{X}} = \sum\nolimits_{j} {s_{ij}^{\text{X}} }$. S^Y = [$s_{jq}^{\text{Y}}$]_m × m is the similarity matrix within Y-view, the degree matrix $D^{\text{Y}} = {\text{diag}}(d_{11}^{\text{Y}} ,d_{22}^{\text{Y}} ,\ldots,d_{nn}^{\text{Y}} )$ with $d_{ii}^{\text{Y}} = \sum\nolimits_{j} {s_{ij}^{\text{Y}} }$. S^XY = [$s_{ij}^{\text{XY}}$]_n × m is the cross-view similarity matrix. D^(XY)r and D^(XY)c are diagonal matrices; their entries are row and column sums of S^XY.

Let $P = \left[ {\begin{array}{*{20}c} {P_{\text{X}} } \\ {P_{\text{Y}} } \\ \end{array} } \right]$, $\varPhi (Z) = \varPhi (X) \oplus \varPhi (Y) = \left[ {\begin{array}{*{20}c} {\varPhi (X)} & {} \\ {} & {\varPhi (Y)} \\ \end{array} } \right]$, $S = \left[ {\begin{array}{*{20}c} {S^{\text{X}} } & {S^{\text{XY}} } \\ {(S^{\text{XY}} )^{\text{T}} } & {S^{\text{Y}} } \\ \end{array} } \right]$, $D = {\text{diag}}(d_{11}^{{}} ,d_{22}^{{}} ,\ldots,d_{(m + n)(m + n)}^{{}} )$ with $d_{ii} = \sum\nolimits_{j} {S_{ij} }$, and $L = D - S$. Then Eq. (20) can be transformed into:

$$\begin{aligned} J(P_{\text{X}} ,P_{\text{Y}} ) & { = }\;{\text{Tr}}\left[ \begin{aligned} P_{\text{X}} \hfill \\ P_{\text{Y}} \hfill \\ \end{aligned} \right]^{\text{T}} \left[ {\begin{array}{*{20}c} {\varPhi (X)} & {} \\ {} & {\varPhi (Y)} \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {D^{\text{X}} + D^{{({\text{XY}})_{\text{r}} }} - S^{\text{X}} } & { - S^{\text{XY}} } \\ { - (S^{\text{XY}} )^{\text{T}} } & {D^{\text{Y}} + D^{{({\text{XY}})_{\text{c}} }} - S^{\text{Y}} } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {\varPhi (X)} & {} \\ {} & {\varPhi (Y)} \\ \end{array} } \right]^{\text{T}} \left[ \begin{aligned} P_{\text{X}} \hfill \\ P_{\text{Y}} \hfill \\ \end{aligned} \right] \\ & = \;{\text{Tr(}}P^{\text{T}} \varPhi (Z)(D - S)\varPhi^{\text{T}} (Z)P )\\ & { = }\;{\text{Tr(}}P^{\text{T}} \varPhi (Z)L\varPhi^{\text{T}} (Z)P ).\\ \end{aligned}$$

(21)