Concept factorization with adaptive graph learning on Stiefel manifold

Abstract

In machine learning and data mining, concept factorization (CF) has achieved great success for its powerful capability in data representation. To learn an adaptive inherent graph structure of data space, and to ease the burden brought by the explicit orthogonality constraint, we propose a concept factorization with adaptive graph learning on the Stiefel manifold (AGCF-SM). The method essentially integrates concept factorization and manifold learning into a unified framework. Therein the adaptive similarity graph is learned by iterative locally linear embedding, which is free from dependence on neighbor sets. An iterative updating algorithm is developed and the convergence and complexity analyses of the algorithm are provided. The numerical experiments on ten benchmark datasets have demonstrated that the proposed algorithm outperforms other state-of-the-art algorithms.

Appendices

Appendix A    The Derivation of (12a) - (12c)

  The derivation includes the following three steps.

1) S-minimization

Fixing W and V , updating S : Optimizing (11) with respect to S is equivalent to solve

$$\begin{aligned} \min _{S}\ \lambda \textrm{Tr}(V^{T} L V)+ \Vert X-XS\Vert _{F}^{2}+\alpha \Vert S\Vert _{F}^{2}+\beta \Vert S\Vert _{1,1},\ \text {s.t.}\ \ S \ge \textbf{0}.~ \end{aligned}$$

(A1)

Let $\mathrm{\Phi } \in \mathbb {R}^{n \times n}$ be the Lagrangian multiplier. Then, the Lagrange function $\mathcal {L}(S, \mathrm{\Phi })$ of (A1) is written as

$$\begin{aligned} \mathcal {L}(S, \mathrm{\Phi })&= \lambda \textrm{Tr}(V^{T} L V)+\Vert X-XS\Vert _{F}^{2}+\alpha \Vert S\Vert _{F}^{2}+\beta \Vert S\Vert _{1,1}-\textrm{Tr}( \mathrm{\Phi }S )\\&= \lambda \textrm{Tr}(TR)+\Vert X-XS\Vert _{F}^{2}+\alpha \textrm{Tr}(S^{T}S)+\beta \textrm{Tr}(\textrm{E} S)-\textrm{Tr}( \mathrm{\Phi } S).~ \end{aligned}$$

For the second equation, $\textrm{Tr}(V^{T}LV)=\frac{1}{2}\sum _{i, j =1}^{n}\Vert V_{i}-V_{j}\Vert ^{2}R_{ij}$, and denote $T_{ij} = \frac{1}{2}\Vert V_{i}-V_{j}\Vert ^{2}$, thus $\textrm{Tr}(V^{T}LV) = \textrm{Tr}(TR)$.

Taking the derivative of $\mathcal {L}(S, \mathrm{\Phi })$ with respect to S and setting it to zero, we have

$$\begin{aligned} \lambda T-2 X^{T} X + 2 X^{T} X S + 2\alpha S+\beta \textrm{E} -\mathrm{\Phi } = \textbf{0}. \end{aligned}$$

Using the KKT condition, $\mathrm{\Phi }_{ij}S_{ij}=0$, we have

$$\begin{aligned} (\lambda T-2 X^{T} X + 2X^{T} X S + 2\alpha S+\beta \textrm{E})_{ij}S_{ij}=0. \end{aligned}$$

Then, we obtain

$$\begin{aligned} S_{ij}^{t+1}\leftarrow S_{ij}^{t}\frac{(X^{T} X)_{ij}}{(\frac{\lambda }{2} T+ X^{T} X S + \alpha S+\frac{\beta }{2}\textrm{E})}_{ij}. \end{aligned}$$

2) V-minimization

Fixing W and S , updating V : Optimizing (11) w.r.t. V is equivalent to solve

$$\begin{aligned} \min _{V} \mathcal {O}_{1}=\Vert X-X W V^{T}\Vert _{F}^{2}+\lambda \textrm{Tr}( V^{T} L V ), \ \text {s.t.} \ V \ge \textbf{0},\ \ V\in \mathcal {M}. \end{aligned}$$

(A2)

As V lies on Stiefel manifold, we compute the natural gradient $\widetilde{\nabla }_{V} \mathcal {O}_{1}$ on Stiefel manifold. Firstly, we calculate the derivative of $\mathcal {O}_{1}$ with respect to V in the Euclidean space,

$$\begin{aligned} \nabla _{V}\mathcal {O}_{1}{} & {} = -2 X^{T} X W +2V W^{T} X^{T} X W+2\lambda L V\\{} & {} \!=\!(2VW^{T} X^{T} X W\!\!+\!\!2\lambda DV)\!-\!(2 X^{T} X W\!\!+\!\!2\lambda RV). \end{aligned}$$

Secondly, we use $\nabla _{V} \mathcal {O}_{1}$ to compute the natural gradient $\widetilde{\nabla }_{V} \mathcal {O}_{1}$ by (9),

$$\begin{aligned} \widetilde{\nabla }_{V}\mathcal {O}_{1}&= \nabla _{V} \mathcal {O}_{1}- V(\nabla _{V} \mathcal {O}_{1})^{T}V \\&\!=\! 2 V W^{T} X^{T} X V \!-\!2 X^{T} X W \!\!+\!\! 2\lambda (D V\!-\! R V \!-\! V V^{T}D^{T} V\!\!+\!\! VV^{T}R^{T}V). \end{aligned}$$

Using the KKT conditions, $V \odot \widetilde{\nabla }_{V} \mathcal {O}_{1} = \textbf{0}$. The updating rule of V is

$$\begin{aligned} V_{jk}^{t+1}\leftarrow V_{jk}^{t} \frac{(X^{T} X W+\lambda (R V+V{V}^{T}{D}^{T}V))_{jk}}{(V{W}^{T} X^{T} X V+\lambda (D V+V{V}^{T} {R}^{T} V))_{jk}}. \end{aligned}$$

3) W-minimization

Fixing S and V , updating W : Optimizing (11) w.r.t. W is equivalent to solve

$$\begin{aligned} \min _{W} \Vert X-X W V^{T}\Vert _{F}^{2}, \ \text {s.t.}\ W \ge \textbf{0}.~ \end{aligned}$$

(A3)

By introducing the Lagrangian multiplier $\mathrm{\Psi }\in \mathbb {R}^{n \times c}$, the Lagrange function $\mathcal {L}(W, \mathrm{\Psi })$ of (A3) is written as

$$\begin{aligned} \mathcal {L}(W, \mathrm{\Psi })= \Vert X-X W V^{T}\Vert _{F}^{2} - \textrm{Tr}(\mathrm{\Psi } W^{T}).~ \end{aligned}$$

Taking the derivative of the Lagrange function $\mathcal {L}(W, \mathrm \Psi )$ with respect to W and setting it to zero, we have

$$\begin{aligned} - 2 X^{T} X V +2 X^{T} X W V^{T} V - \mathrm{\Psi } = \textbf{0}. \end{aligned}$$

Using the KKT conditions, $\mathrm{\Psi }_{jk} W_{jk} = 0$, we have

$$\begin{aligned} (- 2 X^{T} X V +2 X^{T} X W V^{T}V)_{jk} W_{jk}=0. \end{aligned}$$

Then, the updating rule of W is

$$\begin{aligned} W_{jk}^{t+1}\leftarrow W_{jk}^{t}\frac{(X^{T} X V)_{jk}}{(X^{T} X W {V}^{T} V)_{jk}}. \end{aligned}$$

Appendix B    Proof of Theorem 1

In order to prove Theorem 1, we use the property of auxiliary function [44].

Definition 1

$G(h, h^{*})$ is an auxiliary function for F(h) under the conditions

$$G(h, h^{*})\ge F(h),\ \ G(h, h)= F(h).$$

Lemma 1

If $G(h, h^{*})$ is an auxiliary function of F(h), then F(h) is non-increasing under the following updating rule

$$\begin{aligned} h^{t+1} = \mathop {\arg \min }_{h} G(h,h^{t}). \end{aligned}$$

(B1)

Considering the orthogonality of V, the objective function (11) is reformulated as

$$\begin{aligned} F(W,V,S) = \Vert X-X W V^{T}\Vert _{F}^{2} + \lambda \textrm{Tr}(V^{T}LV) + \Vert X-XS\Vert _{F}^{2} \nonumber \\ + \alpha \Vert S\Vert _{F}^{2} + \beta \Vert S\Vert _{1,1}+\textrm{Tr}(\mathrm{\Gamma }(V^{T} V - {I})), \end{aligned}$$

(B2)

where $\mathrm{\Gamma }$ is the Lagrange multiplier.

Denote $F_{S}(S_{i j})$, $F_{V}(V_{j k})$ and $F_{W}(W_{j k})$ as the corresponding objective function only with respect to S, V and W, respectively. Specifically, they are

$$\begin{aligned} F_{S}(S_{i j})= & {} (\lambda T R-2 X^{T} X S + S^{T} X^{T} X S + \alpha S^{T}S + \beta \mathrm{{E}}S)_{i j},\\ F_{V}(V_{j k})= & {} (-2 X^{T} X W V^{T} + V W^{T} X^{T} X W V^{T}+ V^{T} L V+ \mathrm{\Gamma }( V^{T} V - {I} ))_{j k},\\ F_{W}(W_{j k})= & {} (-2 X^{T} X W V^{T}+V W^{T} X^{T} X W V^{T})_{j k}. \end{aligned}$$

To prove the theorem, we construct auxiliary functions for $F_{S}(S_{ij})$, $F_{V}(V_{j k})$ and $F_{W}(W_{j k})$, which are presented as Lemma 2, 3 and 4, respectively.

Lemma 2

Define

$$\begin{aligned} G_{S}(S_{i j}, S_{i j}^{t}) = F_{S}(S_{i j}^{t})&+ F_{S}^{\prime }(S_{i j}^{t}) (S_{i j}-S_{i j}^{t})\nonumber \\&+ \frac{(\frac{\lambda }{2} T+ X^{T} X S^t +\alpha S^t + \frac{\beta }{2} {\mathrm{{E}}})_{i j}}{S_{i j}^{t}}(S_{i j}-S_{i j}^{t})^{2}. \end{aligned}$$

(B3)

Then, $G_{S}(S_{i j}, S_{i j}^{t})$ is an auxiliary function for $F_{S}(S_{i j})$.

Proof

$G_{S}(S_{i j}, S_{i j}) = F_{S}(S_{i j})$ is obvious. Next, we prove $F_{S}(S_{i j}) \le G_{S}(S_{i j}, S_{i j}^{t})$.

The derivatives of the function $F_{S}(S_{ij})$ are

$$\begin{aligned} F^{'}_{S}(S_{ij})&= (\lambda T - 2 X^{T} X + 2 X^{T} X S + 2 \alpha S+\beta \mathrm{{E}})_{ij}, \\ F^{''}_{S}(S_{ij})&= 2 (X^{T} X)_{ii}+2 \alpha ,\\ F^{(k)}_{S}(S_{ij})&= 0, \ k \ge 3. \end{aligned}$$

Therefore, the Taylor series expansion of the function $F_{S}(S_{i j})$ is written as

$$\begin{aligned} F_{S}(S_{i j})= & {} F_{S}(S_{i j}^{t})+F_{S}^{'}(S_{i j}^{t})(S_{i j}-S_{i j}^{t})+\frac{ F_{S}^{''}(S_{i j}^{t})}{2}(S_{i j}-S_{i j}^{t})^{2}\\= & {} F_{S}(S_{i j}^{t})+F_{S}^{\prime }(S_{i j}^{t})(S_{i j}-S_{i j}^{t})+((X^{T} X)_{i i}+\alpha )(S_{i j}-S_{i j}^{t})^{2}.\\ \end{aligned}$$

Compared with (B3), we find that $F_{S}(S_{i j})\le G_{S}(S_{i j}, S_{i j}^{t})$ is equivalent to

$$\begin{aligned} \frac{(\frac{\lambda }{2} T+ X^{T} X S^t +\alpha S^t + \frac{\beta }{2} \mathrm{{E}})_{i j}}{S_{i j}^{t}}\ge (X^{T} X)_{i i}+ \alpha . \end{aligned}$$

It is easy to check that

$$ (X^{T} X S^t)_{ij} = \sum _{\nu }(X^{T} X)_{i \nu }S_{\nu j}^{t}\ge (X^{T} X)_{ii} S_{ij}^{t}.$$

Thus, $F_{S}(S_{i j})\le G_{S}(S_{i j}, S_{i j}^{t})$. $\square$

Lemma 3

Define

$$\begin{aligned} G_{V}(V_{j k}, V_{j k}^{t}) \!\!=\!\! F_{V}(V_{j k}^{t}){} & {} +F_{V}^{\prime }(V_{j k}^{t})(V_{j k}-V_{j k}^{t})\nonumber \\{} & {} \!+\!\frac{(V^t W^{T} X^{T} X W\!+\!\lambda D V^t \!+\! V^t \mathrm{\Gamma }_{1})_{j k}}{V_{j k}^{t}}(V_{j k}\!-\!V_{j k}^{t})^{2}, \end{aligned}$$

(B4)

where

$$\begin{aligned} \begin{array}{l} \mathrm{\Gamma }_{1}\!=\!\mathrm{\Gamma }\!+\!\mathrm{\Gamma }_{2}\!=\!W^{T} X^{T} X V^t\!+\!\lambda (V^t)^{T} R V^t\!-\!W^{T} X^{T} X W ,\\ \mathrm{\Gamma }_{2}=\lambda (V^t)^{T} D V^t \ge 0. \end{array} \end{aligned}$$

(B5)

Then, $G_{V}(V_{j k}, V_{j k}^{t})$ is an auxiliary function for $F_{V}(V_{j k})$.

Proof

Obviously, $F_{V}(V_{j k})=G_{V}(V_{j k}, V_{j k})$ holds. Next, we prove $F_{V}(V_{j k})\le G_{V}(V_{j k}, V_{j k}^{t})$.

To do this, we need to use the Taylor series expansion of the function $F_{V}(V_{j k})$. The derivatives of the function $F_{V}(V_{j k})$ are

$$\begin{aligned} F_{V}^{'}(V_{j k})&\!=\!2(-X^{T} X V + V W^{T} X^{T} X W\!+\!\lambda L V \!+\! \mathrm{\Gamma }V )_{j k},\\ F_{V}^{''}(V_{j k})&=2( (W^{T} X^{T} X W)_{kk}+\lambda L_{jj} + \mathrm{\Gamma }_{kk}),\\ F_{V}^{(k)}(V_{j k})&= 0,\ \ k \ge 3. \end{aligned}$$

Therefore, the Taylor series expansion of the function $F_{V}(V_{j k})$ is

$$\begin{aligned} F_{V}(V_{j k})= & {} F_{V}(V_{j k}^{t}) +F_{V}^{\prime }(V_{j k}^{t})(V_{j k}-V_{j k}^{t})\nonumber \\{} & {} + ((W^{T} X^{T} X W)_{k k}+\lambda L_{j j}+\mathrm{\Gamma }_{k k})(V_{j k}-V_{j k}^{t})^{2}. \end{aligned}$$

Compared with (B4), it is sufficient to show that

$$\begin{aligned} \frac{(V^{t} W^{T} X^{T} X W+\lambda D V^{t}+ V^{t} \mathrm{\Gamma }_{1})_{j k}}{V_{j k}^{t}} \ge (W^{T} X^{T} X W)_{k k}+\lambda L_{j j}+\mathrm{\Gamma }_{k k}. \end{aligned}$$

We obtain the following inequalities,

$$\begin{aligned} (V^{t} W^{T} X^{T} X W)_{j k}= & {} \sum _{\omega } V_{j \omega }^{t}(W^{T} X^{T} X W)_{\omega k} \ge V_{j k}^{t}(W^{T} X^{T} X W)_{k k}, \\ (D V^{t})_{j k}= & {} \sum _{h} D_{j h} V_{h k}^{t} \ge D_{j j} V_{j k}^{t} \ge (D- R )_{j j} V_{j k}^{t}=L_{j j} V_{j k}^{t}, \\ ( V^{t}\mathrm{\Gamma }_{1})_{j k}= & {} \sum _{\tau } V_{j \tau }^{t}(\mathrm{\Gamma }_{1})_{\tau k} \ge V_{j k}^{t} (\mathrm{\Gamma }_{1})_{k k} \ge V_{j k}^{t}(\mathrm{\Gamma }_{1}-\mathrm{\Gamma }_{2})_{k k}=V_{j k}^{t}\mathrm{\Gamma }_{k k}. \end{aligned}$$

Thus, $F_{V}(V_{j k})\le G_{V}(V_{j k}, V_{j k}^{t})$.$\square$

Lemma 4

Define

$$\begin{aligned} G_{W}(W_{j k}, W_{j k}^{t}) = F_{W}(W_{j k}^{t})+F_{W}^{\prime }(W_{j k}^{t})(W_{j k}-W_{j k}^{t})+\frac{( X^{T} X W^t V^{T}V)_{j k}}{W_{j k}^{t}}(W_{j k}-W_{j k}^{t})^{2}. \end{aligned}$$

(B6)

Then, $G_{W}(W_{j k}, W_{j k}^{t})$ is an auxiliary function for $F_{W}(W_{j k})$.

Proof

The proof follows the same line as Lemma 2 and Lemma 3. That is, $G_{W}(W_{j k}, W_{j k}) = F_{W}(W_{j k})$ is obvious, we only need to prove $G_{W}\left( W_{j k}, W_{j k}^{t}\right) \ge F_{W}(W_{j k})$.

To do this, we need to obtain the Taylor series expansion of the function $F_{W}(W_{j k})$. The derivatives of the function $F_{W}(W_{j k})$ are

$$\begin{aligned} F_{W}^{'}(W_{j k})&= 2(-X^{T} X W + X^{T} X W V V^{T})_{j k},\\ F_{W}^{''}(W_{j k})&= 2 (X^{T} X)_{j j}(V V^{T})_{kk},\\ F_{W}^{(k)}(W_{j k})&=0,\ k\ge 3. \end{aligned}$$

Therefore, the Taylor series expansion of the function $F_{W}(W_{j k})$ is obtained as

$$\begin{aligned} F_{W}(W_{j k})= & {} F_{W}(W_{j k}^{t})+F_{W}^{\prime }(W_{j k}^{t})(W_{j k}-W_{j k}^{t})\nonumber \\{} & {} + ( (X^{T} X)_{j j}\ (V V^{T})_{kk})(W_{j k}-W_{j k}^{t})^{2}. \end{aligned}$$

Compared with (B6), we only need to show that

$$\begin{aligned} \frac{(X^{T} X W^t V V^{T})_{j k}}{W_{j k}^{t}} \ge (X^{T} X)_{jj}\ (V V^{T})_{kk}. \end{aligned}$$

(B7)

It is easy to check that

$$\begin{aligned} (X^{T} X W^t V V^{T})_{j k}&=\sum _{l}(X^{T} X W^t)_{j l}(V V^{T})_{l k} \\&\ge (X^{T} X W^t)_{j k}(V V^{T})_{k k} \\&= \sum _{r} (X^{T} X)_{jr} W_{r k}^{t}(V V^{T})_{k k} \\&\ge W_{j k}^{t} (X^{T} X)_{jj}\ (V V^{T})_{kk}. \end{aligned}$$

Thus $G_{W}(W_{j k}, W_{j k}^{t}) \ge F_{W}(W_{j k}).$

Therefore, $G_{W}(W_{j k}, W_{j k}^{t})$ is an auxiliary function for $F_{W}(W_{j k})$.$\square$

Now we give the proof of Theorem 1 based on the above lemmas.

Proof of Theorem

1 According to Lemma 2 - Lemma 4, (B3), (B4) and (B6) are auxiliary functions for $F_{S}(S_{i j})$, $F_{V}(V_{j k})$ and $F_{W}(W_{j k})$ respectively.

Now we prove that the updating rules are exactly the optimal solutions for the auxiliary function. For variable S, by minimizing (B3), we get

$$\begin{aligned} S_{i j}^{t+1}&=\arg \min _{S_{i j}} G_{S}(S_{i j}, S_{i j}^{t})\\&=S_{i j}^{t}-S_{i j}^{t} \frac{F'_{S}(S_{i j}^{t})}{2(\frac{\lambda }{2} T+ X^{T} X S^{t} +\alpha S^{t} + \frac{\beta }{2} \mathrm{{E}})}_{i j} \\&=S_{i j}^{t}\frac{ (X^{T} X)_{i j}}{(\frac{\lambda }{2} T+ X^{T} X S^{t} +\alpha S^{t} + \frac{\beta }{2} \mathrm{{E}})}_{i j} \end{aligned}$$

When $V^{t}$ and $W^{t}$ are fixed, $F_{S}(S_{ij})$ is non-increasing under the rule in (12a).

For variable V, by minimizing (B4), we have

$$\begin{aligned} V_{j k}^{t+1}&=\arg \min _{V_{j k}} G_{V}(V_{j k}, V_{j k}^{t}) \\&=V_{j k}^{t}-V_{j k}^{t}\frac{F'_{V}(V_{j k}^{t})}{2(V^{t} W^{T} X^{T} X S+\lambda D V^{t}+ V^{t} \mathrm{\Gamma }_{1} )_{j k}} \\&=V_{j k}^{t} \frac{(X^{T} X W+\lambda (R V^{t} + V^{t} (V^{t})^{T} D^{T} V^{t}))_{j k}}{(V^{t} W^{T} X^{T} X V^{t}+\lambda (D V^{t}+ V^{t} (V^{t})^{T} R^{T} V^{t}))_{j k}}. \end{aligned}$$

By fixing $W^{t}$ and $S^{t+1}$, $F_{V}(V_{jk})$ is non-increasing under the rule in (12b).

For variable W, by minimizing (B6), we get the updating rule

$$\begin{aligned} \begin{aligned} W_{j k}^{t+1}&=\arg \min _{W_{j k}} G_{W}(W_{j k}, W_{j k}^{t}) \\&=W_{j k}^{t}-W_{j k}^{t} \frac{F_{W}^{'}(W_{j k}^{t})}{2(X^{T} X W^{t} V^{T} V)_{j k}} \\&=W_{j k}^{t} \frac{(X^{T} X V )_{j k}}{( X^{T} X W^{t} V^{T} V)_{j k}}. \end{aligned} \end{aligned}$$

when $V^{t+1}$ and $S^{t+1}$ are fixed, $F_{W}(W_{j k})$ is non-increasing under the rule in (12c).

Therefore, in the t-th iteration, when $V^t$ and $W^t$ are fixed, the following inequality holds

$$ F(W^{t},\ V^t,\ S^{t+1})\le F(W^{t},\ V^t,\ S^t),$$

and when $S^{t+1}$ and $W^t$ are fixed, we have

$$ F(W^{t},\ V^{t+1},\ S^{t+1})\le F(W^{t},\ V^t,\ S^{t+1}),$$

and when $V^{t+1}$ and $S^{t+1}$ are fixed, we get

$$F(W^{t+1},V^{t+1},\ S^{t+1})\le F(W^{t},\ V^{t+1},\ S^{t+1}).$$

As a result, we have

$$\begin{aligned} F(W^{t+1},\ {}&V^{t+1},\ S^{t+1})\le F(W^{t},\ V^{t+1},\ S^{t+1})\le F(W^{t},\ V^{t},\ S^{t+1}) \\&\le F(W^{t},\ V^t,\ S^t)\le \cdots \le F(W^{0},\ V^0,\ S^0). \end{aligned}$$

Therefore, the objective function in (11) is non-increasing under the updating rules (12a), (12b) and (12c).$\square$