Robust automated graph regularized discriminative non-negative matrix factorization

Abstract

Non-negative matrix factorization (NMF) and its variants have been widely employed in clustering and classification task. However, the existing methods do not consider robustness, adaptive graph learning and discrimination information at the same time. To solve this problem, a new nonnegative matrix factorization method is proposed, which is called robust automated graph regularized discriminative non-negative matrix factorization (RAGDNMF). Specifically, L_2,1 norm is used to describe the reconstruction error, the appropriate Laplacian graph is automatically learned and the label information of the training set is added as the regularization term. The ultimate goal is to learn a good projection matrix, which can remove redundant information while preserving the effective components. In addition, we give the multiplicative updating rules for solving optimization problems and convergence proof of objective function. Face recognition experiments on four benchmark datasets show the effectiveness of our proposed method.

Appendix: : (Proof of Theorem)

In the process of theorem proving, the relevant knowledge of matrix calculation can refer to reference book [33]. Due to the use of more knowledge points, this paper will not elaborate in detail for matrix calculation.

In order to prove Theorem, we need to show that O₁ is non-increasing under the updating steps in (25), (26), (27) and (28). For the objective function O₁, we need to fix H, C and A if we update W, so, the first term of O₁ exists. Similarly, we need to fix W, H and A if we update C, the second term of O₁ exists. If we update A, we need to fix W, C and H, the fourth term of O₁ exists. Therefore, we have exactly the same update formula for W, C and A in RAGDNMF as in the original NMF. Thus, we can use the convergence proof of NMF to show that O₁ is nonincreasing under the update step in (25), (27) and (28). These details can be found in [21].

Hence, we only need to prove that O₁ is non-increasing under the updating step in (26). We follow the similar process depicted in [21]. Our proof make use of an auxiliary function and give the definition of the auxiliary function.

Definition 1

$G\left (h, h^{\prime }\right )$ is an auxiliary function of F(h) if the following conditions are satisfied.

$$ G(h,h^{\prime})\geq F(h),~~~~ G(h,h)=F(h) $$

(29)

The above auxiliary function is very important because of the following lemma.

Lemma 1

If G is an auxiliary function of F, then F is non-increasing under the update

$$ h^{(t+1)}=\underset{h}{\arg\min} G\left( h, h^{(t)}\right) $$

(30)

Proof

$F\left (h^{(t+1)}\right )\leq G\left (h^{(t+1)}, h^{(t)}\right )\leq G\left (h^{(t)}, h^{(t)}\right )=F\left (h^{(t)}\right )$

Now, we show that the update step for H in (26) is exactly the update in (30) with a proper auxiliary function.

Considering any element h_ab in H, we use F_ab to denote the part of O₁ which is only relevant to h_ab. It is easy to obtain the following derivatives.

$$ F_{ab}^{\prime}=\left( \frac{\partial O_{1}}{\partial\mathbf{H}}\right)_{ab}=\left[2\mathbf{W}^{T}(\mathbf{W}\mathbf{H}\mathbf{D}-\mathbf{X}\mathbf{D})+2\lambda\mathbf{H}(\mathbf{Q}-\mathbf{P})+2\gamma\mathbf{A}^{T}(\mathbf{A}\mathbf{H}\mathbf{F}-\mathbf{S}\mathbf{F}) \right]_{ab} $$

(31)

$$ F_{ab}^{\prime\prime}=2\left( \mathbf{W}^{T}\mathbf{W}\mathbf{D}\right)_{aa}+2\lambda\mathbf{Q}_{bb}-2\lambda\mathbf{P}_{bb}+2\gamma\left( \mathbf{A}^{T}\mathbf{A}\mathbf{F}\right)_{aa} $$

(32)

It is enough to show that each F_ab is nonincreasing under the update step of (26) because our update is essentially element-wise. Consequently, we introduce the following lemma. □

Lemma 2

Function

$$ \begin{array}{@{}rcl@{}} G\left( h,h_{ab}^{(t)}\right)&=&F_{ab}\left( h_{ab}^{(t)}\right)+F_{ab}^{\prime}\left( h_{ab}^{(t)}\right)\left( h-h_{ab}^{(t)}\right)\\ &&+\frac{\left( \mathbf{W}^{T}\mathbf{W}\mathbf{H}\mathbf{D}+\gamma\mathbf{A}^{T}\mathbf{A}\mathbf{H}\mathbf{F}+\lambda\mathbf{H}\mathbf{Q}\right)_{ab}}{h_{ab}^{(t)}}\left( h-h_{ab}^{(t)}\right)^{2} \end{array} $$

(33)

is an auxiliary function of F_ab.

Proof

We only need to prove that $G\left (h,h_{ab}^{(t)}\right )\geq F_{ab}(h)$ because G(h,h) = F_ab(h) is obvious. Therefore, we first consider the Taylor series expansion of F_ab(h).

$$ \begin{array}{@{}rcl@{}} F_{ab}(h)&=&F_{ab}\left( h_{ab}^{(t)}\right)+F_{ab}^{\prime}\left( h_{ab}^{(t)}\right)\left( h-h_{ab}^{(t)}\right)+\left[\left( \mathbf{W}^{T}\mathbf{W}\mathbf{D}\right)_{aa}\right.\\ &&\left.+\lambda\mathbf{Q}_{bb}-\lambda\mathbf{P}_{bb}+\gamma\left( \mathbf{A}^{T}\mathbf{A}\mathbf{F}\right)_{aa}\right]\left( h-h_{ab}^{(t)}\right)^{2} \end{array} $$

(34)

We compare the (34) with (33) to find that $G\left (h,h_{ab}^{(t)}\right )\geq F_{ab}(h)$ is equivalent to

$$ \begin{array}{ll} \frac{\left( \mathbf{W}^{T}\mathbf{W}\mathbf{H}\mathbf{D}+\gamma\mathbf{A}^{T}\mathbf{A}\mathbf{H}\mathbf{F}+\lambda\mathbf{H}\mathbf{Q}\right)_{ab}}{h_{ab}^{(t)}}\geq \left( \mathbf{W}^{T}\mathbf{W}\mathbf{D}\right)_{aa}+\lambda\mathbf{Q}_{bb}-\lambda\mathbf{P}_{bb}+\gamma\left( \mathbf{A}^{T}\mathbf{A}\mathbf{F}\right)_{aa} \end{array} $$

(35)

In fact, we have

$$ \begin{array}{@{}rcl@{}} \left( \mathbf{W}^{T}\mathbf{W}\mathbf{H}\mathbf{D}+\gamma\mathbf{A}^{T}\mathbf{A}\mathbf{H}\mathbf{F}\right)_{ab}&=&\sum\limits_{q=1}^{r}\left( \mathbf{W}^{T}\mathbf{W}\right)_{aq}h_{qb}^{(t)}\mathbf{D}_{aq} \\ &&+\gamma\sum\limits_{q=1}^{r}\left( \mathbf{A}^{T}\mathbf{A}\right)_{aq}h_{qb}^{(t)}\mathbf{F}_{aq}\geq \left( \mathbf{W}^{T}\mathbf{W}\right)_{aa}h_{ab}^{(t)}\mathbf{D}_{aa} \\ &&+\gamma\left( \mathbf{A}^{T}\mathbf{A}\right)_{aa}h_{ab}^{(t)}\mathbf{F}_{aa} \end{array} $$

(36)

and

$$ \begin{array}{ll} (\lambda\mathbf{H}\mathbf{Q})_{ab}=\lambda\sum\limits_{j=1}^{n}h_{aj}^{(t)}\mathbf{Q}_{jb}\geq \lambda h_{ab}^{(t)}\mathbf{Q}_{bb}\geq\lambda h_{ab}^{(t)}\mathbf{Q}_{bb}-\lambda h_{ab}^{(t)}\mathbf{P}_{bb} \end{array} $$

(37)

Thus, (35) holds and $G\left (h,h_{ab}^{(t)}\right )\geq F_{ab}(h)$. We can now demonstrate the convergence of Theorem: □

Proof of Theorem

Replacing $G\left (h,h_{ab}^{(t)}\right )$ in (30) by (33) results in the following update rule:

$$ \begin{array}{@{}rcl@{}} h_{ab}^{(t+1)}&=&h_{ab}^{(t)}-h_{ab}^{(t)}\frac{F_{ab}^{\prime}\left( h_{ab}^{(t)}\right)}{2\left( \mathbf{W}^{T}\mathbf{W}\mathbf{H}\mathbf{D}+\gamma\mathbf{A}^{T}\mathbf{A}\mathbf{H}\mathbf{F}+\lambda\mathbf{H}\mathbf{Q}\right)_{ab}}\\ &=&h_{ab}^{(t)}\frac{\left( \gamma\mathbf{A}^{T}\mathbf{S}\mathbf{F}+\mathbf{W}^{T}\mathbf{X}\mathbf{D}+\lambda\mathbf{H}\mathbf{P}\right)_{ab}}{\left( \mathbf{W}^{T}\mathbf{W}\mathbf{H}\mathbf{D}+\gamma\mathbf{A}^{T}\mathbf{A}\mathbf{H}\mathbf{F}+\lambda\mathbf{H}\mathbf{Q}\right)_{ab}} \end{array} $$

(38)

Since (33) is an auxiliary function and F_ab is nonincreasing under this update rule.