A Converged Deep Graph Semi-NMF Algorithm for Learning Data Representation

Abstract

Deep nonnegative matrix factorization (DMF) is a particularly useful technique for learning data representation in low-dimensional space. To further obtain the complex hidden information and keep the geometrical structures of the high-dimensional data, we propose a novel deep matrix factorization model with the graph regularization (called DGsnMF). For solving the model with multi-variables, we design a forward–backward splitting scheme. After that, the convergence analysis is attached to the proposed algorithm and it is proved to converge to a critical point. Empirical experiments on benchmark datasets show that the proposed method is superior to the compared methods.

Appendix

To prove Theorem 1, we firstly define the critical points for nonconvex and nonsmooth functions, semi-algebraic functions, and KL functions used for the convergence analysis.

Definition 1

For a proper and lower semi-continuous function h,

• The Fréchet subdifferential of h at x can be written as $\hat{\partial } h(x)$ is a set $\mathbf{u}$ and satisfies the following equation if $x\in$ dom h

  $$\begin{aligned} \left\{ \mathbf{u }\in {\mathbb {R}}^d: \mathop {\lim \inf }\limits _{y\rightarrow x \ y\ne x} \frac{h(y)-h(x)-\langle u,y-x\rangle }{\Vert y-x \Vert }\ge 0\right\} \end{aligned}$$

  (31)

  and $\hat{\partial } h(x)= \emptyset$ when $x\not \in$ dom h

• The limiting subdifferential of h at x, written as $\partial h(x)$, is a set $\mathbf{u}$ defined as follows:

  $$\begin{aligned} \left\{ \mathbf{u }\in {\mathbb {R}}^d: \exists x^k \rightarrow x, h(x^k)\rightarrow h(x), \mathbf{u} ^k \in \hat{\partial } h(x)\rightarrow \mathbf{u} , k \rightarrow \infty \right\} \end{aligned}$$

  (32)

• Point x satisfies the following equation and it is called limiting-critical point of function h:

  $$\begin{aligned} 0 \in \partial h(x) \end{aligned}$$

  (33)

Definition 2

(Kurdyka–Łojasiewicz Property) A function h has the KL property at $x^{*}\in$ dom $\partial h$ if there exist:

• $\eta \in (0,\infty ),s\in (0,\eta )$,a neighborhood V of $x^{*}$

• a continuous concave function $\varphi : [0,\eta )\rightarrow {\mathbb {R}}_{+}$,such that the following holds:

  • a). $\varphi$ is $C^{1}$ on $(0,\eta )$, $\varphi (0)=0$ and $\varphi ^{\prime }(s)>0$

  • b). $\forall x \in V$ satisfies $h(x^{*})<h<h(x^{*}+\eta )$, the KL inequality holds:

    $$\begin{aligned} {\qquad \varphi ^{\prime }\left( h(x)-h\left( x^{*}\right) \right) \mathrm{dist}(0, \partial h(x)) \ge 1} \end{aligned}$$

    (34)

• Function h satisfies the KL property at each dom $\partial h$

Definition 3

(Semi-algebraic sets and functions [2])

• A subset S of ${\mathbb {R}}^n$ is called semi-algebraic set if there exists a finite number of real polynomial functions $g_{ij}$, $g_{ij}^{'}$: ${\mathbb {R}}^n\rightarrow {\mathbb {R}}$ such that

  $$\begin{aligned} S=\bigcup _{j=1}\bigcap _{i=1}\{x\in {\mathbb {R}}^n:g_{ij}(x)=0, g_{ij}^{'}(x)<0\} \end{aligned}$$

  (35)

• A function $h: {\mathbb {R}}^n \rightarrow (-\infty ,+\infty ]$ is semi-algebraic if its graph $\{(x,y)\in {\mathbb {R}}^n\times {\mathbb {R}},y=h(x)\}$ is a semi-algebraic set.

Secondly, following the conditions as described in Lemma 1, we check the sequence $\{\mathbf{J }^{k}\} =\{\mathbf{W }_{1}^{k},\mathbf{W} _{2}^{k},\ldots ,\mathbf{W} _{l}^{k}, \mathbf{H} _{l}^{k}\}$ generated by Algorithm 1 satisfies the conditions V1-V4.

Proof

(Condition V1): Forward–backward splitting framework is to solve the DGsnMF problem as described in (9). As for weighted matrix $\mathbf{W} _{i}\ (i \in 1,2,\ldots ,l)$, it is updated through the iteration as described in (19) and we have

$$\begin{aligned}&\frac{\mu _{i}^{k}}{2}\left\| \mathbf{W }_{i}^{k+1} -\left( \mathbf{W} _{i}^{k}-\frac{1}{\mu _{i}^{k}} \nabla _\mathbf{W _{i}}Q(\mathbf{W} _{1}^{k+1},\ldots , \mathbf{W} _{i}^{k},\ldots , \mathbf{W} _{l}^{k}, \mathbf{H} _{l}^{k})\right) \right\| _{F}^{2} +F(\mathbf{W} _{i}^{k+1})\nonumber \\&\quad \le \frac{\mu _{i}^{k}}{2}\Vert \frac{1}{\mu _{i}^{k}} \nabla _\mathbf{W _{i}}Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k},\ldots , \mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k}) \Vert _{F}^{2}+F(\mathbf{W} _{i}^{k}) \end{aligned}$$

(36)

then,

$$\begin{aligned} F(\mathbf{W} _{i}^{k})&\ge F(\mathbf{W} _{i}^{k+1}) +\frac{\mu _{i}^{k}}{2}\Vert \mathbf{W }_{i}^{k+1} -\mathbf{W} _{i}^{k}\Vert _{F}^{2}\nonumber \\&\quad +Tr\langle \mathbf{W} _{i}^{k+1}-\mathbf{W} _{i}^{k}, \nabla _\mathbf{W _{i}}Q(\mathbf{W} _{1}^{k+1},\ldots , \mathbf{W} _{i}^{k},\ldots , \mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k})\rangle . \end{aligned}$$

(37)

It is noteworthy that $\nabla _\mathbf{W _{i}}Q$ is Lipschitz continuous and Q is the $C^1$ function. Let $L_\mathbf{W _{i}}$ denote the Lipschitz constant of gradient $\nabla _\mathbf{W _{i}}Q$ and we have

$$\begin{aligned}&Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k+1},\ldots , \mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k})\le Q (\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k},\ldots ,\mathbf{W} _{l}^{k}, \mathbf{H} _{l}^{k})\nonumber \\&\quad + Tr\langle \mathbf{W} _{i}^{k+1} -\mathbf{W} _{i}^{k},\nabla _\mathbf{W _{i}}Q(\mathbf{W} _{1}^{k+1},\ldots , \mathbf{W} _{i}^{k},\ldots , \mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k})\rangle \nonumber \\&\qquad + \frac{L_\mathbf{W _{i}}}{2}\Vert \mathbf{W }_{i}^{k+1} -\mathbf{W} _{i}^{k}\Vert _{F}^{2}. \end{aligned}$$

(38)

Combining (37) and (38), we obtain the following inequality:

$$\begin{aligned}&F(\mathbf{W} _{i}^{k+1})+Q(\mathbf{W} _{1}^{k+1},\ldots , \mathbf{W} _{i}^{k+1},\ldots ,\mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k}) \le F(\mathbf{W} _{i}^{k})\nonumber \\&\quad + Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k},\ldots , \mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k}) - \frac{\mu _{i}^{k} -L_\mathbf{W _{i}}}{2}\Vert \mathbf{W }_{i}^{k+1}-\mathbf{W} _{i}^{k}\Vert _{F}^{2}. \end{aligned}$$

(39)

Similarly, as for feature matrix $\mathbf{H} _{l}$, we have,

$$\begin{aligned}&Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k+1},\ldots , \mathbf{W} _{l}^{k+1},\mathbf{H} _{l}^{k+1}) +G(\mathbf{H} _{l}^{k+1}) \le G(\mathbf{H} _{i}^{k})\nonumber \\&\quad + Q(\mathbf{W} _{1}^{k+1},\mathbf{W} _{2}^{k+1},\ldots , \mathbf{W} _{l}^{k+1},\mathbf{H} _{l}^{k}) - \frac{\nu ^{k} -L_\mathbf{H _{l}}}{2}\Vert \mathbf{H }_{l}^{k+1}-\mathbf{H} _{l}^{k}\Vert _{F}^{2}. \end{aligned}$$

(40)

The step sizes sequences $\mu _{i}^{k}$ and $\nu ^{k}$ are chosen as $0 < \frac{1}{\mu _{i}^{k}} \le \frac{1}{L_{Q}}$ and $0<\frac{1}{\nu ^{k}} \le \frac{1}{L_{Q}}$. Summing up (39) and (40), we can thereby obtain the following equation:

$$\begin{aligned}&\phi (\mathbf{W} _{1}^{k},\mathbf{W} _{2}^{k},\ldots ,\mathbf{W} _{l}^{k}, \mathbf{H} _{l}^{k})-\phi (\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k+1}, \ldots ,\mathbf{W} _{l}^{k+1},\mathbf{H} _{l}^{k+1})\nonumber \\&\quad \ge \sum _{i=1}^{l}\frac{\mu _{i}^{k}-L_\mathbf{W _{i}}}{2} \Vert \mathbf{W }_{i}^{k+1}-\mathbf{W} _{i}^{k}\Vert _{F}^{2} +\frac{\nu ^{k} -L_\mathbf{H _{l}}}{2}\Vert \mathbf{H }_{l}^{k+1}-\mathbf{H} _{l}^{k}\Vert _{F}^{2}. \end{aligned}$$

(41)

Let $a_{1}=(\mu _{i}^{k}-L_\mathbf{W _{i}})/2$ and $a_{2}=(\nu ^{k}-L_\mathbf{H _{l}})/2$. There exists $a:=(a_{1},a_{2}) > 0$ since ${\mu _{i}^{k}} >{L_{Q}}$ and ${\nu ^{k}} >{L_{Q}}$. Thus, $\{\mathbf{J }^{k}\}=\{\mathbf{W }_{1}^{k}, \mathbf{W} _{2}^{k},\ldots ,\mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k}\}$ satisfy the condition V1. $\square$

Proof

(Condition V2): The iteration of weighted matrix $\mathbf{W} _{i}$ is presented as follows:

$$\begin{aligned} \mathbf{W} _{i}^{k+1}=\mathop {\arg \min }\limits _\mathbf{W _{i}} F(\mathbf{W} _{i})+ \hat{Q}(\mathbf{W} _{i})+\frac{\mu _{i}^{k}}{2} \Vert \mathbf{W }_{i}-\mathbf{W} _{i}^{k+1}\Vert _{F}^{2} \end{aligned}$$

(42)

where

$$\begin{aligned} \hat{Q}(\mathbf{W} _{i})&=Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k},\ldots , \mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k})\nonumber \\&\quad +Tr\langle \mathbf{W} _{i} -\mathbf{W} _{i}^{k+1}, \nabla _\mathbf{H _{l}}Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k},\ldots , \mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k})\rangle . \end{aligned}$$

(43)

Therefore, we have

$$\begin{aligned} 0\in \partial F(\mathbf{W} _{i}^{k+1})+ \nabla _\mathbf{W _{i}} Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k},\ldots ,\mathbf{W} _{l}^{k}, \mathbf{H} _{l}^{k})+\mu _{i}^{k}(\mathbf{W} _{i}^{k+1}-\mathbf{W} _{i}^{k}). \end{aligned}$$

(44)

Then,

$$\begin{aligned}&\nabla _\mathbf{W _{i}}Q(\mathbf{J} ^{k+1})-\nabla _\mathbf{W _{i}} Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k},\ldots ,\mathbf{W} _{l}^{k}, \mathbf{H} _{l}^{k})\nonumber \\&\qquad -\mu _{i}^{k}(\mathbf{W} _{i}^{k+1}-\mathbf{W} _{i}^{k}) \in \partial _\mathbf{W _{i}}\phi (\mathbf{J} ^{k+1}). \end{aligned}$$

(45)

For the feature matrix $\mathbf{H} _{l}$, the iteration has been presented in (16), we can obtain:

$$\begin{aligned}&\nabla _\mathbf{H _{l}}Q(\mathbf{J} ^{k+1})-\nabla _\mathbf{H _{l}} Q(\mathbf{W} _{1}^{k+1},\ldots ,\mathbf{W} _{i}^{k+1},\ldots ,\mathbf{W} _{l}^{k+1}, \mathbf{H} _{l}^{k})\nonumber \\&\quad -\nu ^{k}(\mathbf{H} _{l}^{k+1}-\mathbf{H} _{l}^{k}) \in \partial _\mathbf{H _{l}}\phi (\mathbf{J} ^{k+1}). \end{aligned}$$

(46)

Define: $\mathbf{N} ^{k}:=(\mathbf{N} _\mathbf{W _{i}}^{k}, \mathbf{N} _\mathbf{H _{l}}^{k})$

$$\begin{aligned} \mathbf{N} _\mathbf{W _{i}}^{k+1}&: = \nabla _\mathbf{W _{i}} Q(\mathbf{J} ^{k+1})-\nabla _\mathbf{W _{i}}Q(\mathbf{W} _{1}^{k+1},\ldots , \mathbf{W} _{i}^{k},\ldots ,\mathbf{W} _{l}^{k},\mathbf{H} _{l}^{k})\nonumber \\&\quad -\mu _{i}^{k}(\mathbf{W} _{i}^{k+1}-\mathbf{W} _{i}^{k}) \in \partial _\mathbf{W _{i}}\phi (\mathbf{J} ^{k+1}) \end{aligned}$$

(47)

$$\begin{aligned} \mathbf{N} _\mathbf{H _{l}}^{k+1}&: = \nabla _\mathbf{H _{l}}Q (\mathbf{J} ^{k+1})-\nabla _\mathbf{H _{l}}Q(\mathbf{W} _{1}^{k+1},\ldots , \mathbf{W} _{i}^{k+1},\ldots ,\mathbf{W} _{l}^{k+1},\mathbf{H} _{l}^{k})\nonumber \\&\quad -\nu ^{k}(\mathbf{H} _{l}^{k+1}-\mathbf{H} _{l}^{k}) \in \partial _\mathbf{H _{l}}\phi (\mathbf{J} ^{k+1}). \end{aligned}$$

(48)

If $\{\mathbf{J }^{k}\}$ is bounded, since $\nabla Q$ is Lipschitz continuous on any bounded set. Then, $\mathbf{N} ^{k} :=(\mathbf{N} _\mathbf{W _{i}}^{k},\mathbf{N} _\mathbf{H _{l}}^{k})\in \partial \phi (\mathbf{J} ^{k})$, exist positive constants $p>0$ and satisfy the following inequality:

$$\begin{aligned} \Vert \mathbf{N }^{k+1}\Vert _{F}^{2}\le p\Vert \mathbf{J }^{k+1} -\mathbf{J} ^{k} \Vert _{F}^{2} \end{aligned}$$

(49)

and Condition V2 is proved. $\square$

Proof

(Condition V3): From V1, $\phi (\mathbf{J} )$ is decreasing and $inf \{\phi \}>-\infty$. Given a positive integer j, we can obtain the following inequality:

$$\begin{aligned} \phi (\mathbf{J} ^{0})-\phi (\mathbf{J} ^{j})\ge a\sum _{k=0}^{j} \Vert \mathbf{J }^{k}-\mathbf{J} ^{k+1}\Vert _{F}^{2}. \end{aligned}$$

(50)

Therefore, there exists some $\phi (\tilde{\mathbf{J }})$ such that $\phi (\mathbf{J} ^{j})\rightarrow \phi (\tilde{\mathbf{J }})$ as $j\rightarrow +\infty$. When $j\rightarrow +\infty$, we have

$$\begin{aligned} \sum _{k=0}^{j\rightarrow +\infty } \Vert \mathbf{J }^{k}-\mathbf{J} ^{k+1} \Vert _{F}^{2}< \phi (\mathbf{J} ^{0})-\phi (\mathbf{J} ^{j})<+\infty \end{aligned}$$

(51)

which means $\text {lim}\Vert \mathbf{J }^{k}-\mathbf{J} ^{k+1} \Vert _{F}^{2}=0$.

For weight matrices $\mathbf{W} _{i}$, from (42), we assume $k= k_{j}-1(j\rightarrow +\infty )$ and $\mathbf{W} _{i} =\tilde{\mathbf{W }}_{i}$ and obtain

$$\begin{aligned}&F(\mathbf{W} _{i}^{k_{j}})+ \hat{Q}(\mathbf{W} _{i}^{k_{j}}) +\frac{\mu _{i}^{k_{j}-1}}{2}\Vert \mathbf{W }_{i}^{k_{j}} -\mathbf{W} _{i}^{k_{j}-1}\Vert _{F}^{2}\nonumber \\&\quad \le F(\tilde{\mathbf{W }}_{i})+ \hat{Q}(\tilde{\mathbf{W }}_{i}) +\frac{\mu _{i}^{k_{j}-1}}{2}\Vert \tilde{\mathbf{W }}_{i} -\mathbf{W} _{i}^{k_{j}-1}\Vert _{F}^{2}. \end{aligned}$$

(52)

From Condition V1 and Lipschitz continuity of $\nabla Q$, we can obtain

$$\begin{aligned} \text {lim}\ \text {sup}_{j\rightarrow +\infty } F(\mathbf{W} _{i}^{k_{j}}) \le F(\tilde{\mathbf{W }}_{i}). \end{aligned}$$

(53)

Similarly, for feature matrix $\mathbf{H} _{l}$, we get

$$\begin{aligned} \text {lim}\ \text {sup}_{j\rightarrow +\infty } G(\mathbf{H} _{l}^{k_{j}}) \le G(\tilde{\mathbf{H }}_{l}). \end{aligned}$$

(54)

It should be noted that function F and G are lower semi-continuous, we have,

$$\begin{aligned} \text {lim}_{j\rightarrow +\infty } F(\mathbf{W} _{i}^{k_{j}}) =F(\tilde{\mathbf{W }}_{i}), \text {lim}_{j\rightarrow +\infty } G(\mathbf{H} _{l}^{k_{j}})=G(\tilde{\mathbf{H }}_{l}). \end{aligned}$$

(55)

Moreover, Q is continuous, we can thereby obtain the following equation:

$$\begin{aligned} \text {lim}_{j\rightarrow +\infty } \phi (\mathbf{J} ^{k_{j}}) =\phi (\tilde{\mathbf{J }}), \end{aligned}$$

(56)

and complete the proof of Condition V3. $\square$

Proof

(Condition V4): Q is a real polynomial function, so it is naturally a semi-algebraic function. The graph of indicator function $\delta _{S}$ can be rewritten as $\{(u,0):u\in S\}\cup \{(v,0):v\in \bar{S}\}$, set $S_{F}$ and $S_{G}$ are both nonempty closed semi-algebraic sets according to their definitions, therefore, functions F and G are semi-algebraic . Our objective function $\phi (\mathbf{W} _{1},\ldots ,\mathbf{W} _{l},\mathbf{H} _{l}) =Q(\mathbf{W} _{1},\ldots ,\mathbf{W} _{l},\mathbf{H} _{l}) +I(\mathbf{W} _{1},\ldots ,\mathbf{W} _{l},\mathbf{H} _{l})$ defined in (9) satisfies the KL property. $\square$