Unified and Coupled Self-Stabilizing Algorithms for Minor and Principal Eigen-pairs Extraction

Abstract

Neural network algorithms on principal component analysis (PCA) and minor component analysis (MCA) are of importance in signal processing. Unified (dual purpose) algorithm is capable of both PCA and MCA, thus it is valuable for reducing the complexity and the cost of hardware implementations. Coupled algorithm can mitigate the speed-stability problem which exists in most noncoupled algorithms. Though unified algorithm and coupled algorithm have these advantages compared with single purpose algorithm and noncoupled algorithm, respectively, there are only few of unified algorithms and coupled algorithms have been proposed. Moreover, to the best of the authors’ knowledge, there is no algorithm which is both unified and coupled has been proposed. In this paper, based on a novel information criterion, we propose two self-stabilizing algorithms which are both unified and coupled. In the derivation of our algorithms, it is easier to obtain the results compared with traditional methods, because it is not needed to calculate the inverse Hessian matrix. Experiment results show that the proposed algorithms perform better than existing coupled algorithms and unified algorithms.

Appendices

Appendix 1: Derivation of Coupled System

Suppose that $\bar{\mathbf{W}}$ contains all eigenvectors of C in its columns $(\mathbf{w}_1, \ldots ,\mathbf{w}_n)$, coincides with eigenvalues $\lambda _1 < \cdots < \lambda _n$. $\bar{\varvec{\Lambda }}$ is a diagonal matrix containing all eigenvalues of C, which is, $\bar{\varvec{\Lambda }}=\text {diag} (\lambda _1, \ldots , \lambda _n)$ . Suppose that $\mathbf{e}_1=(1,0,\ldots ,0)^T$, then it holds that $\mathbf{C}=\bar{\mathbf{W}}\bar{\varvec{\Lambda }}\bar{\mathbf{W}}^T$, $\bar{\mathbf{W}}{} \mathbf{e}_1=\mathbf{w}_1$ and $\bar{\mathbf{W}}^{-1}=\bar{\mathbf{W}}^T$ [11]. We have

$$\begin{aligned} \mathbf{C}-\lambda \mathbf{I}=\bar{\mathbf{W}} \left( \bar{\varvec{\Lambda }} -\lambda \mathbf{I}\right) \bar{\mathbf{W}}^T. \end{aligned}$$

(59)

In the vicinity of the stationary point $(\mathbf{w}_1, \lambda _1)$, by approximating $\mathbf{w} \approx \mathbf{w}_1, \lambda \approx \lambda _1 \ll \lambda _j\ (2 \le j \le n)$, it holds that

$$\begin{aligned} \bar{\varvec{\Lambda }}-\lambda \mathbf{I} \approx \bar{\varvec{\Lambda }}-\lambda \mathbf{e}_1 \mathbf{e}_1^T. \end{aligned}$$

(60)

Then we have

$$\begin{aligned} \mathbf{C} - \lambda \mathbf{I} \approx \bar{\mathbf{W}}\left( {\bar{\mathbf \Lambda }} - \lambda \mathbf{e}_1 \mathbf{e}_1^T \right) \bar{ \mathbf{W}}^T\approx \mathbf{C} - \lambda \mathbf{w}{} \mathbf{w}^T . \end{aligned}$$

(61)

In this case, (10) and (11) change to

$$\begin{aligned} \left( \mathbf{C} - \lambda \mathbf{w}{} \mathbf{w}^T \right) \dot{\mathbf{w}} - \mathbf{w}\dot{\lambda }&= - \left( \mathbf{C} - \lambda \mathbf{w}{} \mathbf{w}^T \right) \mathbf{w} \end{aligned}$$

(62)

$$\begin{aligned} -2\mathbf{w}^T\dot{\mathbf{w}}&= \mathbf{w}^T\mathbf{w} - 1, \end{aligned}$$

(63)

which equals to

$$\begin{aligned} \mathbf{C}\dot{\mathbf{w}} - \lambda \mathbf{w}{} \mathbf{w}^T\dot{\mathbf{w}} - \mathbf{w}\dot{\lambda }&= - \mathbf{C}{} \mathbf{w} + \lambda \mathbf{w}{} \mathbf{w}^T \mathbf{w} \end{aligned}$$

(64)

$$\begin{aligned} \mathbf{w}^T\dot{\mathbf{w}}&= \frac{1}{2}(1-\mathbf{w}^T\mathbf{w}). \end{aligned}$$

(65)

Substituting (65) into (64), it yields

$$\begin{aligned} \mathbf{C}\dot{\mathbf{w}} - \mathbf{w}\dot{\lambda }= - \mathbf{C}{} \mathbf{w} + \frac{1}{2}\lambda \mathbf{w}\left( \mathbf{w}^T \mathbf{w}+1\right) . \end{aligned}$$

(66)

Premultiplying both side of (66) by $\mathbf{w}^T\mathbf{C}^{-1}$, then we get

$$\begin{aligned} \mathbf{w}^T\dot{\mathbf{w}} - \mathbf{w}^T \mathbf{C}^{-1}{} \mathbf{w}\dot{\lambda }= - \mathbf{w}^T\mathbf{w} + \frac{1}{2}\lambda \mathbf{w}^T \mathbf{C}^{-1}{} \mathbf{w}\left( \mathbf{w}^T \mathbf{w}+1\right) . \end{aligned}$$

(67)

Substituting (65) into (67) and after some manipulations, we have that

$$\begin{aligned} \dot{\lambda } = \frac{\mathbf{w}^T \mathbf{w}+1}{2}\left( \frac{1}{\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}-\lambda \right) . \end{aligned}$$

(68)

Substituting (68) into (66), we can get

$$\begin{aligned} \dot{\mathbf{w}} = \frac{ \left( \mathbf{w}^T \mathbf{w}+1 \right) \mathbf{C}^{-1}{} \mathbf{w}}{2\mathbf{w}^T\mathbf{C}^{-1} \mathbf{w}}-\mathbf{w}. \end{aligned}$$

(69)

Then, the coupled dynamical system (12)–(13) is obtained.

Next we analyze the approximation error of the coupled system. It is found that

$$\begin{aligned} \big |(\bar{\varvec{\Lambda }}-\lambda \mathbf{I}) - (\bar{\varvec{\Lambda }}-\lambda \mathbf{e}_1 \mathbf{e}_1^T)\big |=\big |-\lambda \mathbf{I} +\lambda \mathbf{e}_1 \mathbf{e}_1^T\big |=\lambda (\mathbf{I}- \mathbf{e}_1 \mathbf{e}_1^T), \end{aligned}$$

(70)

and hence

$$\begin{aligned} \bar{\mathbf{W}}[ \lambda (\mathbf{I} - \mathbf{e}_1 \mathbf{e}_1^T)]\bar{ \mathbf{W}}^T = \lambda (\mathbf{I}- \mathbf{w}{} \mathbf{w}^T). \end{aligned}$$

(71)

Substituting (71) into (10)–(11), we have that

$$\begin{aligned} \lambda (\mathbf{I}- \mathbf{w}{} \mathbf{w}^T)\Delta \dot{\mathbf{w}} - \mathbf{w}\Delta \dot{\lambda }&= - \lambda (\mathbf{I}- \mathbf{w}{} \mathbf{w}^T)\mathbf{w} \end{aligned}$$

(72)

$$\begin{aligned} -2\mathbf{w}^T\Delta \dot{\mathbf{w}}&= \mathbf{w}^T\mathbf{w} - 1, \end{aligned}$$

(73)

where $\Delta \dot{\mathbf{w}}$ and $\Delta \dot{\lambda }$ denote the approximation error of $\dot{\mathbf{w}}$ and $\dot{\lambda }$, respectively. Similarly, we can get that

$$\begin{aligned} \Delta \dot{\mathbf{w}}&= \frac{1}{2}{} \mathbf{w}\left( \frac{1}{\mathbf{w}^T\mathbf{w}}-1\right) \end{aligned}$$

(74)

$$\begin{aligned} \Delta \dot{\lambda }&= \frac{1}{2}\lambda \left( \frac{1}{\mathbf{w}^T\mathbf{w}}-\mathbf{w}^T\mathbf{w}\right) \end{aligned}$$

(75)

Since convergence analysis (see Sect. 4) shows that the coupled system (12)–(13) has the only stable stationary point $(\mathbf{w}_1,\lambda _1)$ and it holds that $\mathbf{w}^T_1\mathbf{w}_1=1$. Substituting $\mathbf{w}^T\mathbf{w}\approx 1$ into (74)–(75), it holds that $\Delta \dot{\mathbf{w}}\approx \mathbf{0}$ and $\Delta \dot{\lambda }\approx 0$. This means that the approximation error approaches to 0 in the vicinity of the stable stationary point.

Appendix 2: The Jacobian of fMCA

The Jacobian of coupled rule is defined as

$$\begin{aligned} \mathbf{J}(\mathbf{w},\lambda )=\left( \begin{array}{c|c} {\frac{\partial \dot{\mathbf{w}}}{\partial \mathbf{w}^T}} &{} {\frac{\partial \dot{\mathbf{w}}}{\partial \lambda }} \\ {\frac{\partial \dot{\lambda }}{\partial \mathbf{w}^T}} &{} {\frac{\partial \dot{\lambda }}{\partial \lambda }} \end{array} \right) . \end{aligned}$$

(76)

For fMCA, we have

$$\begin{aligned} \frac{\partial \dot{\mathbf{w}}}{\partial \mathbf{w}^T}&=\frac{\partial }{\partial \mathbf{w}^T}\frac{\mathbf{C}^{-1}{} \mathbf{w}{} \mathbf{w}^T\mathbf{w}}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}+\frac{\partial }{\partial \mathbf{w}^T}\frac{\mathbf{C}^{-1}{} \mathbf{w}}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}-\frac{\partial }{\partial \mathbf{w}^T}{} \mathbf{w}\nonumber \\&=\mathbf{A}(\mathbf{w})+\mathbf{B}(\mathbf{w})-\mathbf{I}, \end{aligned}$$

(77)

where

$$\begin{aligned} \mathbf{A}(\mathbf{w})&=\frac{\partial }{\partial \mathbf{w}^T}\frac{\mathbf{C}^{-1}{} \mathbf{w}{} \mathbf{w}^T\mathbf{w}}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}\nonumber \\&=\frac{\mathbf{C}^{-1}}{2} \frac{\partial }{\partial \mathbf{w}^T} \frac{\mathbf{w}{} \mathbf{w}^T\mathbf{w}}{\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}\nonumber \\&=\frac{\mathbf{C}^{-1}}{2} \left( \frac{\mathbf{w}^T\mathbf{w}{} \mathbf{I}+2\mathbf{w}{} \mathbf{w}^T}{\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}- \frac{2\mathbf{w}{} \mathbf{w}^T\mathbf{w}{} \mathbf{w}^T\mathbf{C}^{-1}}{(\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w})^2}\right) \nonumber \\&=\frac{\mathbf{C}^{-1}{} \mathbf{w}^T\mathbf{w}}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}+\frac{\mathbf{C}^{-1}{} \mathbf{w}{} \mathbf{w}^T}{\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}} -\frac{\mathbf{C}^{-1}{} \mathbf{w}{} \mathbf{w}^T\mathbf{w}{} \mathbf{w}^T\mathbf{C}^{-1}}{(\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w})^2} \end{aligned}$$

(78)

and

$$\begin{aligned} \mathbf{B}(\mathbf{w})&=\frac{\partial }{\partial \mathbf{w}^T}\frac{\mathbf{C}^{-1}{} \mathbf{w}}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}\nonumber \\&=\frac{\mathbf{C}^{-1}}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}} - \frac{4\mathbf{C}^{-1}{} \mathbf{w}{} \mathbf{w}^T\mathbf{C}^{-1}}{(2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w})^2}\nonumber \\&=\frac{\mathbf{C}^{-1}}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}-\frac{\mathbf{C}^{-1}{} \mathbf{w}{} \mathbf{w}^T\mathbf{C}^{-1}}{(\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w})^2}. \end{aligned}$$

(79)

Similarly,

$$\begin{aligned} \frac{\partial \dot{\lambda }}{\partial \mathbf{w}^T}&=\frac{\partial }{\partial \mathbf{w}^T} \frac{\mathbf{w}^T\mathbf{w}+1}{2\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}- \frac{\partial }{\partial \mathbf{w}^T}\frac{\lambda }{2}\left( \mathbf{w}^T\mathbf{w}+1\right) \nonumber \\&=\frac{\mathbf{w}^T}{\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w}}-\frac{\left( \mathbf{w}^T\mathbf{w}+1\right) \mathbf{w}^T\mathbf{C}^{-1}}{(\mathbf{w}^T\mathbf{C}^{-1}{} \mathbf{w})^2} -\lambda \mathbf{w}^T. \end{aligned}$$

(80)

In the vicinity of the stationary point $(\mathbf{w}_1,\lambda _1)$, it holds that $\mathbf{C}^{-1}{} \mathbf{w}_1\approx \lambda ^{-1}_1\mathbf{w}_1$, $\mathbf{w}^T_1\mathbf{w}_1\approx 1$ and $\mathbf{w}^T_1\mathbf{C}^{-1}{} \mathbf{w}_1\approx \lambda ^{-1}_1$. Then

$$\begin{aligned} \mathbf{A}(\mathbf{w})&\approx \frac{1}{2}\lambda _1\mathbf{C}^{-1}+\mathbf{w}_1\mathbf{w}^T_1-\mathbf{w}_1\mathbf{w}^T_1=\frac{1}{2}\lambda _1\mathbf{C}^{-1} \end{aligned}$$

(81)

$$\begin{aligned} \mathbf{B}(\mathbf{w})&\approx \frac{1}{2}\lambda _1\mathbf{C}^{-1}-\mathbf{w}_1\mathbf{w}^T_1. \end{aligned}$$

(82)

Thus

$$\begin{aligned} \frac{\partial \dot{\mathbf{w}}}{\partial \mathbf{w}^T}=\lambda _1\mathbf{C}^{-1}-\mathbf{w}_1\mathbf{w}^T_1-\mathbf{I}, \end{aligned}$$

(83)

and

$$\begin{aligned} \frac{\partial \dot{\lambda }}{\partial \mathbf{w}^T}=-2\lambda _1\mathbf{w}^T. \end{aligned}$$

(84)

It is also found that

$$\begin{aligned} \frac{\partial \dot{\mathbf{w}}}{\partial \lambda }=0 \end{aligned}$$

(85)

and

$$\begin{aligned} \frac{\partial \dot{\lambda }}{\partial \lambda }=-\frac{\mathbf{w}^T\mathbf{w}+1}{2}=-1. \end{aligned}$$

(86)

To sum up, we get

$$\begin{aligned} \mathbf{J}(\mathbf{w}_1,\lambda _1)=\left( \begin{array}{c|c} {\mathbf{C}^{-1}\lambda _1-\mathbf{I}-\mathbf{w}_1\mathbf{w}_1^T} &{} {\mathbf{0}} \\ {-2\lambda _1\mathbf{w}_1^T} &{} {-1} \end{array} \right) . \end{aligned}$$

(87)

Appendix 3: Proof of $\lambda (k) > 0$

The proof of aMCA, fMCA and aPCA are similar, thus we only take aMCA as an example. Suppose that C has the smallest eigenvalue $\lambda _1$ and the largest eigenvalue $\lambda _n$. We have proved that $\mathbf{w}(k)$ is self-stabilized, or in other words, the norm of $\mathbf{w}(k)$ converges to 1 automatically. In this case, we can approximate $\mathbf{w}^T\mathbf{w}=1$ in each step. We have

$$\begin{aligned} \lambda _1\le \frac{\mathbf{w}^T\mathbf{C}{} \mathbf{w}}{\mathbf{w}^T\mathbf{w}}=\mathbf{w}^T\mathbf{C}{} \mathbf{w}\le \lambda _n, \end{aligned}$$

(88)

which is the property of Rayleigh quotient [17]. From (23) we conclude that $\mathbf{Q}=\mathbf{C}^{-1}$ has the smallest eigenvalue $1/\lambda _n$ and the largest eigenvalue $1/\lambda _1$. Thus

$$\begin{aligned} \frac{1}{\lambda _n}\le \frac{\mathbf{w}^T\mathbf{Q}{} \mathbf{w}}{\mathbf{w}^T\mathbf{w}}=\mathbf{w}^T\mathbf{Q}{} \mathbf{w}\le \frac{1}{\lambda _1}, \end{aligned}$$

(89)

or in other words

$$\begin{aligned} \lambda _1\le \frac{1}{\mathbf{w}^T\mathbf{Q}{} \mathbf{w}}\le \lambda _n. \end{aligned}$$

(90)

Since $\lambda (0)>0$ and $0<\gamma (k)<1$ hold for all $k \ge 0$, we can verify (21) as

$$\begin{aligned} \lambda (k+1)&=(1-\gamma (k))\lambda (k)+\gamma (k)\frac{1}{\mathbf{w}^T(k)\mathbf{Q}(k)\mathbf{w}(k)}\nonumber \\&\ge (1-\gamma (k))\lambda (k)+\gamma (k)\lambda _1\nonumber \\&\ge \cdots \nonumber \\&\ge \prod _{i=0}^k(1-\gamma (i))\lambda (0)+\lambda _1\sum _{i=0}^k\left[ \gamma (i)\prod _{j=i+1}^k(1-\gamma (j))\right] \nonumber \\&> 0 \end{aligned}$$

(91)

for all $k\ge 0$.