Non-negative Matrix Factorization with Pairwise Constraints and Graph Laplacian

Abstract

Non-negative matrix factorization (NMF) is a very effective method for high dimensional data analysis, which has been widely used in information retrieval, computer vision, and pattern recognition. NMF aims to find two non-negative matrices whose product approximates the original matrix well. It can capture the underlying structure of data in the low dimensional data space using its parts-based representations. However, NMF is actually an unsupervised method without making use of prior information of data. In this paper, we propose a novel pairwise constrained non-negative matrix factorization with graph Laplacian method, which not only utilizes the local structure of the data by graph Laplacian, but also incorporates pairwise constraints generated among all labeled data into NMF framework. More specifically, we expect that data points which have the same class label will have very similar representations in the low dimensional space as much as possible, while data points with different class labels will have dissimilar representations as much as possible. Consequently, all data points are represented with more discriminating power in the lower dimensional space. We compare our approach with other typical methods and experimental results for image clustering show that this novel algorithm achieves the state-of-the-art performance.

Appendix

In this section, we prove the convergence of PCGNMF. We begin with the following theorem regarding the iterative updating rules in Eqs. (15) and (16).

Theorem 1

The objective function ${J}$ is nonincreasing under the iterative updating rules in Eqs. (15) and (16). The objective function is invariant under these updates if and only if $\mathbf {U}$ and $\mathbf {V}$ are at a stationary point.

Theorem 1 guarantees that these iterative updating rules of $\mathbf {U}$ and $\mathbf {V}$ in Eqs. (15) and (16) can converge on a stationary point and hence final solution will be a local optimum. To prove Theorem 1, we have to show that $J$ is nonincreasing under the iterative updating rules in Eqs. (15) and (16). Since the second term and the third term of $J$ are only related to $\mathbf {V}$, and the iterative updating rule (16) is exactly the same as update formula for $\mathbf {U}$ in the NMF. The convergence proof of NMF has shown that $J$ is nonincreasing under the iterative updating rule in Eq. (16) [11]. So, we only need to prove that $J$ is nonincreasing under the iterative updating rule in Eq. (15). Firstly, we make use of a similar auxiliary function which has been used in the Expectation-Maximization algorithm [5, 21].

Definition

G $(v, v')$ is an auxiliary function for F($v$) if the conditions G $(v, v')$ $\ge$ F($v$), G $(v, v)$ = F($v$) are satisfied.

We have the following lemma regarding the very useful auxiliary function, which will be helpful to prove the convergence of the objective function.

Lemma 1

If G is an auxiliary function of F, then F is nonincreasing under the update

$$\begin{aligned} v^{(t+1)} = \arg \min \limits _{v} \textit{G}(v, v^t) \end{aligned}$$

(20)

Proof

F $(v^{(t+1)})$ $\le$ G $(v^{(t+1)},v^{t})$ $\le$ G $(v^{t},v^{t})=$ F $(v^{t})$

Now, we will prove that the iterative updating rule for $\mathbf {V}$ in Eq. (15) is exactly the update rule in Eq. (20) with an appropriate auxiliary function. For any entry $v_{ab}$ in $\mathbf {V}$, we use $F_{v_{ab}}$ to denote the part of $J$ only relevant to $v_{ab}$. It is easy to check that

$$\begin{aligned} \textit{F}'_{v_{ab}}&= (\frac{\partial J }{\partial {\mathbf {V}}})_{ab} =-2(\mathbf {X}^T\mathbf {U})_{ab}+2(\mathbf {V}\mathbf {U}^T\mathbf {U})_{ab} + 2\alpha (\mathbf {D}\mathbf {V})_{ab} \nonumber \\&-2\alpha (\mathbf {W}\mathbf {V})_{ab} + \beta (\mathbf {M}\mathbf {V}\mathbf {A}+\mathbf {C}\mathbf {V})_{ab}\end{aligned}$$

(21)

$$\begin{aligned} \textit{F}''_{v_{ab}}&= 2(\mathbf {U}^T\mathbf {U})_{bb} + 2\alpha \mathbf {D}_{aa} - 2\alpha \mathbf {W}_{aa} + \beta \mathbf {C}_{aa} \end{aligned}$$

(22)

Where $\textit{F}'$, $\textit{F}''$ are the first and second order derivative with respect to $\mathbf {V}$, respectively. $\square$

Lemma 2

The function

$$\begin{aligned} \textit{G}(v,v^{(t)}_{ab})&= \textit{F}_{v_{ab}}(v^{(t)}_{ab})+\textit{F}'_{v_{ab}}(v^{(t)}_{ab})(v-v^{(t)}_{ab}) \nonumber \\&+\frac{(\mathbf {V}\mathbf {U}^T\mathbf {U})_{ab} + \alpha (\mathbf {D}\mathbf {V})_{ab} + \frac{\beta }{2}(\mathbf {M}\mathbf {V}\mathbf {A}+\mathbf {C}\mathbf {V})_{ab}}{v^{(t)}_{ab}} (v-v^{(t)}_{ab})^2 \end{aligned}$$

(23)

is an auxiliary function for $F_{v_{ab}}$, and it is the part of $J$ related $v_{ab}$.

Proof

Since $\textit{G}(v,v)=F_{v_{ab}}(v)$ is explicit, we only have to show that $\textit{G}(v,v^{(t)}_{ab})\ge F_{v_{ab}}(v)$. In order to achieve that, we can compare the Taylor series expansion of $F_{v_{ab}}(v)$ with the auxiliary function $\textit{G}(v,v^{(t)}_{ab})$.

$$\begin{aligned} \textit{F}_{v_{ab}}(v)&= \textit{F}_{v_{ab}}(v^{(t)}_{ab})+\textit{F}'_{v_{ab}}(v^{(t)}_{ab})(v-v^{(t)}_{ab}) \\&+\left[ (\mathbf {U}^T\mathbf {U})_{bb} + \alpha \mathbf {D}_{aa} - \alpha \mathbf {W}_{aa} + \frac{\beta }{2} \mathbf {C}_{aa}\right] (v-v^{(t)}_{ab})^2 \nonumber \end{aligned}$$

(24)

Clearly, showing $\textit{G}(v,v^{(t)}_{ab})\ge F_{v_{ab}}(v)$ is equivalent to prove that

$$\begin{aligned} \frac{(\mathbf {V}\mathbf {U}^T\mathbf {U})_{ab} + \alpha (\mathbf {D}\mathbf {V})_{ab} + \frac{\beta }{2} (\mathbf {M}\mathbf {V}\mathbf {A}+\mathbf {C}\mathbf {V})_{ab}}{v^{(t)}_{ab}}&\ge (\mathbf {U}^T\mathbf {U})_{bb} + \alpha \mathbf {D}_{aa}\\&- \alpha \mathbf {W}_{aa} + \frac{\beta }{2} \mathbf {C}_{aa} \nonumber \end{aligned}$$

(25)

In order to prove above inequality holds, we have

$$\begin{aligned} (\mathbf {V}\mathbf {U}^T\mathbf {U})_{ab}=\sum _{c=1}^k v_{ac}^{(t)}(\mathbf {U}^T\mathbf {U})_{cb} \ge v_{ab}^{(t)}(\mathbf {U}^T\mathbf {U})_{bb} \end{aligned}$$

(26)

and

$$\begin{aligned} \alpha (\mathbf {D}\mathbf {V})_{ab}&= \alpha \sum _{j=1}^n \mathbf {D}_{aj}v_{jb}^{(t)} \ge \alpha \mathbf {D}_{aa} v_{ab}^{(t)}\end{aligned}$$

(27)

$$\begin{aligned} \frac{\beta }{2} (\mathbf {C}\mathbf {V})_{ab}&= \frac{\beta }{2} \sum _{j=1}^n \mathbf {C}_{aj}v_{jb}^{(t)} \ge \frac{\beta }{2} v_{ab}^{(t)}\mathbf {C}_{aa} \end{aligned}$$

(28)

Therefore, the inequality $\textit{G}(v,v^{(t)}_{ab})\ge F_{v_{ab}}(v)$ holds.$\square$

Now, we can show the convergence of Theorem 1 for $\mathbf {V}$:

Proof of Theorem 1

we can replace $\textit{G}(v,v^{(t)}_{ab})$ in Eq. (20) by Eq. (23) to obtain the update rule which is exactly the same as the iterative updating rule for $\mathbf {V}$.

$$\begin{aligned} v_{ab}^{(t+1)}&= v_{ab}^{(t)}\frac{[2\mathbf {V}\mathbf {U}^T\mathbf {U}+ 2\alpha \mathbf {DV} + \beta (\mathbf {M}\mathbf {V}\mathbf {A}+\mathbf {C}\mathbf {V})]_{ab}-\textit{F}'_{v_{ab}}(v^{(t)}_{ab})}{[2\mathbf {V}\mathbf {U}^T\mathbf {U}+ 2\alpha \mathbf {DV} + \beta (\mathbf {M}\mathbf {V}\mathbf {A}+\mathbf {C}\mathbf {V})]_{ab}}\\&= v_{ab}^{(t)}\frac{2(\mathbf {X}^T\mathbf {U})_{ab} + 2\alpha (\mathbf {W}\mathbf {V})_{ab} }{2(\mathbf {V}\mathbf {U}^T\mathbf {U})_{ab} + 2\alpha (\mathbf {DV})_{ab} + \beta (\mathbf {M}\mathbf {V}\mathbf {A}+\mathbf {C}\mathbf {V})_{ab}}\nonumber \end{aligned}$$

(29)

Since Eq. (23) is an auxiliary function, $\textit{F}_{v_{ab}}$ is nonincreasing under this updating rule with Lemma 2.$\square$