Hyper-graph regularized discriminative concept factorization for data representation

Abstract

For the tasks of pattern analysis and recognition, nonnegative matrix factorization and concept factorization (CF) have attracted much attention due to its effective application to find the meaningful low-dimensional representation of data. However, they neglect the geometry information embedded in the local neighborhoods of the data and fail to exploit the prior knowledge. In this paper, a novel semi-supervised learning algorithm named hyper-graph regularized discriminative concept factorization (HDCF) is proposed. For the sake of exploring intrinsic geometrical structure of the data and making use of label information, HDCF incorporates hyper-graph regularizer into CF framework and uses the label information to train a classifier for the classification task. HDCF can learn a new concept factorization with respect to the intrinsic manifold structure of the data and also simultaneously adapted to the classification task and a classifier built on the low-dimensional representations. Moreover, an iterative updating optimization scheme is developed to solve the objective function of the proposed HDCF and the convergence proof of our optimization scheme is also provided. Experimental results on ORL, Yale and USPS image databases demonstrate the effectiveness of our proposed algorithm.

Appendix A (proof of theorem 1)

To prove Theorem 1, we need to show that the objective function ${\varvec{J}}_\mathbf{HDCF }$ in Eq. (12) is nonincreasing under the updating rules stated in Eqs. (20), (21) and (23). Now, we make use of an auxiliary function similar to that used in the EM algorithm (Dempster et al. 1977) to prove the convergence of Theorem 1. We begin with the definition of the auxiliary function.

Definition 1

The function $G( {x,{x}'} )$ is an auxiliary function for F(x), if the $G( {x,{x}'} )\ge F( x )$ and $G( {x,x} )=F(x )$ are satisfied.

The auxiliary function is very useful because of the following lemma.

Lemma 1

If G is an auxiliary function of F, then F is nonincreasing under the update

$$\begin{aligned} x^{(t+1)}=\mathop {\arg \min }\limits _{x} G({x,x^{(t)}} ) \end{aligned}$$

(28)

Proof

$$\begin{aligned} F( {x^{(t+1)}} )\le G( {x^{(t+1)},x^{(t)}} )\le G( {x^{(t)},x^{(t)}} )=F( {x^{(t)}} ). \end{aligned}$$

Since the updating rule of ${\varvec{W}}$ is exactly the same with the original CF, the convergence proof of Eq. (20) can be referred to Xu and Gong (2004). Here we only need to prove the convergence of the updating rules for ${\varvec{V}}$ and ${\varvec{A}}$ in Eqs. (21) and (23). Next we will show that the updating rule for ${\varvec{V}}$ in Eq. (21) is exactly the update in Eq. (28) with a proper auxiliary function.

Considering any element $v_{ab}$ in ${\varvec{V}}$, we use $F_{v_{ab} }$ to denote the part of ${\varvec{J}}_\mathbf{HDCF }$ which is only relevant to $v_{ab}$. It is easy to check that

$$\begin{aligned} {F'}_{v_{ab} }= & {} \left( {\frac{\partial {\varvec{J}}_\mathbf{HDCF } }{\partial {\varvec{V}}}} \right) _{ab} \\= & {} \left( {-2{\varvec{KW}}+2{\varvec{VW}}^{T} {\varvec{KW}}+2\alpha {\varvec{L}}^\mathrm{hyper}{\varvec{V}}} \right) _{ab}\nonumber \\&+\,2\beta \left( {-{\varvec{C}}^{T}{\varvec{Y}}^{T} {\varvec{A}}+{\varvec{C}}^{T}{\varvec{CVA}}^{T}{\varvec{A}}} \right) _{ab}, \\ {F}_{v_{ab}}^{\prime \prime }= & {} 2({{\varvec{W}}^{T}{\varvec{KW}}} )_{bb} +2\alpha \left( {{\varvec{L}}^{\mathrm{hyper}}} \right) _{aa} \nonumber \\&+\,2\beta ({\varvec{C}}^{T}{\varvec{C}})_{aa} ( {\varvec{A}}^{T}{\varvec{A}})_{bb} \end{aligned}$$

Since our update is essentially elementwise, it is sufficient to show that each $F_{v_{ab} }$ is nonincreasing under the update step of Eq. (21). $\square$

Lemma 2

The function in Eq. (2) is an auxiliary function for $F_{v_{ab} }$.

$$\begin{aligned} G(v,v_{ab}^{(t)} )= & {} F_{v_{ab} } (v_{ab}^{(t)} )+{F}'_{v_{ab} } (v_{ab}^{(t)} )(v-v_{ab}^{(t)} )\nonumber \\&+\frac{({\varvec{VW}}^{T}{\varvec{KW}})_{ab} +\alpha ({\varvec{D}}_{v} {\varvec{V}})_{ab} +\beta ({\varvec{C}}^{T}{\varvec{CVA}}^{T}{\varvec{A}})_{ab} }{v_{ab}^{(t)} }\nonumber \\&(v-v_{ab}^{(t)} )^{2} \end{aligned}$$

(29)

Proof

Since $G(v,v)=F_{v_{ab} } (v)$ is obvious, we need show that $G(v,v_{ab}^{(t)} )\ge F_{v_{ab} } (v)$. To do this, we compare the Taylor series expansion of $F_{v_{ab} } (v)$

$$\begin{aligned} F_{v_{ab} } (v)= & {} F_{v_{ab} } (v_{ab}^{(t)} )+{F}'_{v_{ab} } (v_{ab}^{(t)} )(v-v_{ab}^{(t)} )\\&+\,[({\varvec{W}}^{T}{\varvec{KW}})_{bb} +\alpha ({\varvec{L}}^{\mathrm{hyper}})_{aa} \\&+\,\beta ({\varvec{C}}^{T}{\varvec{C}})_{aa} ({\varvec{A}}^{T}{\varvec{A}})_{bb} ](v-v_{ab}^{(t)} )^{2} \end{aligned}$$

with Eq. (28) to find that $G(v,v_{ab}^{(t)} )\ge F_{v_{ab} } (v)$ is equivalent to

$$\begin{aligned}&\frac{({\varvec{VW}}^{T}{\varvec{KW}})_{ab} +\alpha ({\varvec{D}}_{v} {\varvec{V}})_{ab} +\beta ({\varvec{C}}^{T}{\varvec{CVA}}^{T}{\varvec{A}})_{ab} }{v_{ab}^{(t)} }\nonumber \\&\quad \ge ({\varvec{W}}^{T}{\varvec{KW}})_{bb} +\alpha ({\varvec{L}}^{\mathrm{hyper}})_{aa} +\beta ({\varvec{C}}^{T}{\varvec{C}})_{aa} ({\varvec{A}}^{T}{\varvec{A}})_{bb}\nonumber \\ \end{aligned}$$

(30)

We have

$$\begin{aligned} ({{\varvec{VW}}^{T}{\varvec{KW}}})_{ab}= & {} \sum \limits _{q=1}^r {v_{aq}^{(t)} \left( {{\varvec{W}}^{T}{\varvec{KW}}} \right) _{qb} } \ge v_{ab}^{(t)} \left( {{\varvec{W}}^{T}{\varvec{KW}}} \right) _{bb} ;\\ \alpha ({\varvec{D}}_{v} {\varvec{V}})_{ab}= & {} \alpha \sum \limits _{j=1}^n {({\varvec{D}}_{v} )_{aj} v_{jb}^{(t)} } \ge \alpha ({\varvec{D}}_{v} )_{aa} v_{ab}^{(t)}\\\ge & {} \alpha ({\varvec{D}}_{v} -{\varvec{S}})_{aa} v_{ab}^{(t)} =\alpha ({\varvec{L}}^\mathrm{hyper})_{aa} v_{ab}^{(t)} \end{aligned}$$

And

$$\begin{aligned} \beta ({\varvec{C}}^{T}{\varvec{CVA}}^{T}{\varvec{A}})_{ab}= & {} \beta \sum \limits _{q=1}^r {\left( {{\varvec{C}}^{T}{\varvec{CV}}} \right) _{aq} \left( {{\varvec{A}}^{T}{\varvec{A}}} \right) _{qb} } \\\ge & {} \beta \left( {{\varvec{C}}^{T}{\varvec{CV}}} \right) _{ab} \left( {{\varvec{A}}^{T}{\varvec{A}}} \right) _{bb} \\\ge & {} \beta \sum \limits _{j=1}^n {\left( {{\varvec{C}}^{T}{\varvec{C}}} \right) _{aj} v_{jb}^{(t)} \left( {{\varvec{A}}^{T}{\varvec{A}}} \right) _{bb} } \\\ge & {} \beta v_{ab}^{\left( t \right) } \left( {{\varvec{C}}^{T}{\varvec{C}}} \right) _{aa} \left( {{\varvec{A}}^{T}{\varvec{A}}} \right) _{bb} \end{aligned}$$

Thus, Eq. (30) holds and $G(v,v_{ab}^{(t)} )\ge F_{v_{ab} } (v)$. $\square$

Next we define an auxiliary function for the update rule in Eq. (23). Similarly, consider any element $a_{ab}$ in A; we use $F_{a_{ab} }$ to denote the part of ${\varvec{J}}({\varvec{A}})$ which is only relevant to $a_{ab}$. It is easy to check that

$$\begin{aligned} {F'}_{a_{ab} }= & {} \left( {\frac{\partial {\varvec{J}}({\varvec{A}})}{\partial {\varvec{A}}}} \right) _{ab} \\= & {} \left( {-2{\varvec{YCV}}+2{\varvec{AV}}^{T}{\varvec{C}}^{T} {\varvec{CV}}+2\gamma {\varvec{A}}} \right) _{ab} , \\ {{{F}''}}_{a_{{{ab}}} }= & {} 2({{\varvec{V}}^{T}{\varvec{C}}^{T}{\varvec{CV}}})_{bb} +2\gamma \end{aligned}$$

Similarly, it is sufficient to show that each $F_{a_{ab} }$ is nonincreasing under the update step of Eq. (23). Then the auxiliary function regarding $a_{ab}$ is defined as follows:

Lemma 3

The function in Eq. (31) is an auxiliary function for $F_{a_{ab}}$.

$$\begin{aligned} G(a,a_{ab}^{(t)} )= & {} F_{a_{ab} } (a_{ab}^{(t)} )+{F}'_{a_{ab} } (a_{ab}^{(t)} )(a-a_{ab}^{(t)} )\nonumber \\&+\frac{\left( {{\varvec{AV}}^{T}{\varvec{C}}^{T}{\varvec{CV}}+\gamma {\varvec{A}}} \right) _{ab} }{a_{ab}^{(t)} }(a-a_{ab}^{(t)} )^{2} \end{aligned}$$

(31)

Proof

Since $G(a,a)=F_{a_{ab} } (a)$ is obvious, we need show that $G(a,a_{ab}^{(t)} )\ge F_{a_{ab} } (a)$. To do this, we compare the Taylor series expansion of $F_{a_{ab} } (a)$

$$\begin{aligned} F_{a_{ab} } (v)= & {} F_{a_{ab} } (a_{ab}^{(t)} )+{F}'_{a_{ab} } (a_{ab}^{(t)} )\left( a-a_{ab}^{(t)} \right) \\&+\,[( {{\varvec{V}}^{T}{\varvec{C}}^{T}{\varvec{CV}}} )_{bb} +\gamma ]\left( a-a_{ab}^{(t)} \right) ^{2} \end{aligned}$$

With Eq. (28) to find that $G(a,a_{ab}^{(t)} )\ge F_{a_{ab} } (a)$ is equivalent to

$$\begin{aligned} \frac{\left( {{\varvec{AV}}^{T}{\varvec{C}}^{T}{\varvec{CV}}+\gamma {\varvec{A}}} \right) _{ab} }{a_{ab}^{(t)} }\ge [( {{\varvec{V}}^{T}{\varvec{C}}^{T}{\varvec{CV}}})_{bb} +\gamma ] \end{aligned}$$

(32)

We have $({\varvec{AV}}^{T}{\varvec{C}}^{T}{\varvec{CV}})_{ab} =\sum \limits _{q=1}^r {a_{aq}^{(t)} ({\varvec{V}}^{T}{\varvec{C}}^{T}{\varvec{CV}})_{qb} } \ge a_{ab}^{(t)}$ $({\varvec{V}}^{T}{\varvec{C}}^{T}{\varvec{CV}})_{bb}$;

$$\begin{aligned} \gamma ({{\varvec{A}}})_{{\varvec{ab}}}= & {} \gamma ( {{\varvec{AI}}})_{{\varvec{ab}}} \\= & {} \gamma \sum \nolimits _{q=1}^r {a_{aq}^{(t)} ({\varvec{I}})_{qb} } \ge \gamma a_{ab}^{(t)} ({\varvec{I}})_{bb}\\= & {} \gamma a_{ab}^{(t)} \end{aligned}$$

Thus, Eq. (32) holds and $G(a,a_{ab}^{(t)} )\ge F_{a_{ab} } (a)$. $\square$

Now we can demonstrate the convergence of Theorem 1:

Proof of Theorem 1

Replacing $G(v,v_{ab}^{(t)} )$ in Eq. (28) by Eq. (20), we get

$$\begin{aligned} v_{ab}^{(t+1)}= & {} v_{ab}^{(t)} -v_{ab}^{(t)} \frac{{F}'_{v_{ab} } (v_{ab}^{(t)} )}{2({\varvec{VW}}^{T}{\varvec{KW}})_{ab} +2\alpha ({\varvec{D}}_{v} {\varvec{V}})_{ab} +2\beta ({\varvec{C}}^{T}{\varvec{CVA}}^{T}{\varvec{A}})_{ab} }\\= & {} v_{ab}^{(t)} \frac{\left( {{\varvec{KW}}+\alpha {\varvec{SV}}+\beta {\varvec{C}}^{T}{\varvec{Y}}^{T}{\varvec{A}}} \right) _{ab} }{\left( {{\varvec{VW}}^{T}{\varvec{KW}}+\alpha {\varvec{D}}_{v} {\varvec{V}}+\beta {\varvec{C}}^{T}{\varvec{CVA}}^{T}{\varvec{A}}} \right) _{ab} } \end{aligned}$$

Since Eq. (20) is an auxiliary function, $F_{v_{ab} }$ is nonincreasing under this updating rule.

Similarly, replacing $G(a,a_{ab}^{(t)} )$ in Eq. (28) by Eq. (23), we get

$$\begin{aligned} a_{ab}^{(t+1)}= & {} a_{ab}^{(t)} -a_{ab}^{(t)} \frac{{F}'_{a_{ab} } (a_{ab}^{(t)} )}{2({\varvec{AV}}^{T}{\varvec{C}}^{T}{\varvec{CV}}+\gamma {\varvec{A}})_{ab} }\\= & {} a_{ab}^{(t)} \frac{\left( {{\varvec{YCV}}} \right) _{ab} }{\left( {{\varvec{AV}}^{T}{\varvec{C}}^{T}{\varvec{CV}}+\gamma {\varvec{A}}} \right) _{ab} } \end{aligned}$$

Since Eq. (22) is an auxiliary function, $F_{a_{ab} }$ is nonincreasing under this updating rule. $\square$