Graph-Regularized Local Coordinate Concept Factorization for Image Representation

Abstract

Existing matrix factorization based techniques, such as nonnegative matrix factorization and concept factorization, have been widely applied for data representation. In order to make the obtained concepts to be as close to the original data points as possible, one state-of-the-art method called locality constraint concept factorization is put forward, which represent the data by a linear combination of only a few nearby basis concepts. But its locality constraint does not well reveal the intrinsic data structure since it only requires the concept to be as close to the original data points as possible. To address these problems, by considering the manifold geometrical structure in local concept factorization via graph-based learning, we propose a novel algorithm, called graph-regularized local coordinate concept factorization (GRLCF). By constructing a parameter-free graph using constrained Laplacian rank (CLR) algorithm, we also present an extension of GRLCF algorithm as $\hbox {GRLCF}_{\mathrm{CLR}}$. Moreover, we develop the iterative updating optimization schemes, and provide the convergence proof of our optimization scheme. Since GRLCF simultaneously considers the geometric structures of the data manifold and the locality conditions as additional constraints, it can obtain more compact and better structured data representation. Experimental results on ORL, Yale and Mnist image datasets demonstrate the effectiveness of our proposed algorithm.

Appendix: (Proof of Theorem 1)

To prove Theorem 1, we need to show that the objective function ${\varvec{J}}_{\varvec{GRLCF}}$ in Eq. (13) is nonincreasing under the updating rules stated in Eqs. (16) and (17). Now, we make use of an auxiliary function similar to that used in the EM algorithm [31] to prove the convergence of the theorem 1. We begin with the definition of the auxiliary function.

Definition 2

The function $G\left( {x,{x}'} \right)$ is an auxiliary function for F(x), if the $G\left( {x,{x}'} \right) \ge F(x)$ and $G\left( {x,x} \right) =F(x)$ are satisfied.

The auxiliary function is very useful because of the following lemma.

Lemma 1

if G is an auxiliary function of F, then F is nonincreasing under the update

$$\begin{aligned} x^{(t+1)}=\mathop {\arg \min }\limits _{x} G\left( {x,x^{(t)}} \right) \end{aligned}$$

(27)

Proof

$F\left( {x^{(t+1)}} \right) \le G\left( {x^{(t+1)},x^{(t)}} \right) \le G\left( {x^{(t)},x^{(t)}} \right) =F\left( {x^{(t)}} \right)$.

Next we will show that the updating rule for ${\varvec{V}}$ in Eq. (17) is exactly the update in Eq. (27) with a proper auxiliary function.

Considering any element $v_{ab}$ in ${\varvec{V}}$, we use $F_{v_{ab} }$ to denote the part of ${\varvec{J}}_{\varvec{GRLCF}}$ which is only relevant to $v_{ab}$. It is easy to check that

$$\begin{aligned} {{F}'}_{v_{{ab}} }= & {} \left( {\frac{\partial {\varvec{J}}_{{\varvec{GRLCF}}} }{\partial {\varvec{V}}}} \right) _{{ab}} \\= & {} \left[ {2{\varvec{W}}^{{\varvec{T}}}{\varvec{KWV}}-2\,{\varvec{W}}^{{\varvec{T}}}{\varvec{K}}+\lambda \left( {{\varvec{A}}-2\;\,{\varvec{W}}^{{\varvec{T}}}{\varvec{K}}+{\varvec{B}}} \right) +2\,\;\mu \,{\varvec{VL}}} \right] _{{ab}} ,\\ {{F''}}_{v_{ab} }= & {} 2({\varvec{W}}^{{{\varvec{T}}}}{\varvec{KW}})_{aa} +2\mu {\varvec{L}}_{bb} \end{aligned}$$

Since our update is essentially element-wise, it is sufficient to show that each $F_{v_{ab} }$ is nonincreasing under the update step of Eq. (17).

Lemma 2

Function

$$\begin{aligned} G\left( v,v_{ab}^{(t)} \right)= & {} F_{v_{ab} } \left( v_{ab}^{(t)} \right) +{F}'_{v_{ab} } \left( v_{ab}^{(t)}\right) \left( v-v_{ab}^{(t)} \right) \nonumber \\&+\frac{({\varvec{W}}^{T}\varvec{KWV}+\textstyle {1 \over 2}\lambda {\varvec{A}}+\textstyle {1 \over 2}\lambda {\varvec{B}}+\mu {{\varvec{V}}}{{\varvec{E}}})_{ab} }{v_{ab}^{(t)} }(v-v_{ab}^{(t)} )^{2} \end{aligned}$$

(28)

is an auxiliary function for $F_{v_{ab} }$.

Proof

Since $G(v,v)=F_{v_{ab} } (v)$ is obvious, we need show that $G(v,v_{ab}^{(t)} )\ge F_{v_{ab} } (v)$. To do this, we compare the Taylor series expansion of $F_{v_{ab} } (v)$

$$\begin{aligned} F_{v_{ab} } (v)= & {} F_{v_{ab} } (v_{ab}^{(t)} )+{F}'_{v_{ab} } (v_{ab}^{(t)} )(v-v_{ab}^{(t)} )\\&+[({\varvec{W}}^{T}{{\varvec{K}}}{{\varvec{W}}})_{aa} +\mu {\varvec{L}}_{bb} ](v-v_{ab}^{(t)} )^{2} \end{aligned}$$

with Eq. (27) to find that $G(v,v_{ab}^{(t)} )\ge F_{v_{ab} } (v)$ is equivalent to

$$\begin{aligned} \frac{({\varvec{W}}^{T}\varvec{KWV}+{1 \over 2}\lambda {\varvec{A}}+\textstyle {1 \over 2}\lambda {\varvec{B}}+\mu {{\varvec{V}}}{{\varvec{E}}})_{ab} }{v_{ab}^{(t)} }\ge ({\varvec{W}}^{T}{{\varvec{K}}}{{\varvec{W}}})_{aa} +\mu {\varvec{L}}_{bb} \end{aligned}$$

(29)

From the definition of ${\varvec{A}}$ and ${\varvec{B}}$, it is easy to check that ${\varvec{A}}\ge 0$ and ${\varvec{B}}\ge 0$. Thus we have

$$\begin{aligned} ({\varvec{W}}^{T}\varvec{KWV}+\textstyle {1 \over 2}\lambda {\varvec{A}}+\textstyle {1 \over 2}\lambda {\varvec{B}}+\mu {{\varvec{V}}}{{\varvec{E}}})_{ab}= & {} ({\varvec{W}}^{T}\varvec{KWV}+\textstyle {1 \over 2}\lambda {\varvec{A}}+\textstyle {1 \over 2}\lambda {\varvec{B}})_{ab} +\mu \sum \limits _{i=1}^N {v_{ai}^{(t)} {\varvec{E}}_{ib} }\\\ge & {} ({\varvec{W}}^{T}\varvec{KWV})_{ab} +\mu \sum \limits _{i=1}^N {v_{ai}^{(t)} {\varvec{E}}_{ib} } \\= & {} \sum \limits _k {({\varvec{W}}^{T}{{\varvec{K}}}{{\varvec{W}}})_{ak} {\varvec{V}}_{kb} } +\mu \sum \limits _{i=1}^N {v_{ai}^{(t)} {\varvec{E}}_{ib} }\\\ge & {} v_{ab} ({\varvec{W}}^{T}{{\varvec{K}}}{{\varvec{W}}})_{aa} +\mu v_{ab} {\varvec{E}}_{bb}\\\ge & {} v_{ab} ({\varvec{W}}^{T}{{\varvec{K}}}{{\varvec{W}}})_{aa} +\mu v_{ab} ({\varvec{E}}-{\varvec{S}})_{bb} \\= & {} v_{ab} ({\varvec{W}}^{T}{{\varvec{K}}}{{\varvec{W}}})_{aa} +\mu v_{ab} {\varvec{L}}_{bb} \end{aligned}$$

Thus, Eq. (29) holds and $G(v,v_{ab}^{(t)} )\ge F_{v_{ab} } (v)$.

Next we define an auxiliary function for the update rule in Eq. (16). Similarly, consider any element $w_{ab}$ in ${\varvec{W}}$, we use $F_{w_{ab} }$ to denote the part of ${\varvec{J}}_{\varvec{GRLCF}}$ which is only relevant to $w_{ab}$. Then the auxiliary function regarding $w_{ab}$ is defined as follows:

Lemma 3

Function

$$\begin{aligned} G(w,w_{ab}^{(t)} )= & {} F_{w_{ab} } (w_{ab}^{(t)} )+{F}'_{w_{ab} } (w_{ab}^{(t)} )(w-w_{ab}^{(t)} )\nonumber \\&+\frac{\left( {\varvec{KWVV}^{T}} \right) _{ab}+\lambda \left( {\sum \nolimits _{i=1}^N {\varvec{KWD}_{i} } } \right) _{ab} }{w_{ab}^{(t)} }(w-w_{ab}^{(t)} )^{2} \end{aligned}$$

(30)

is an auxiliary function for $F_{w_{ab}}$.

The proof of Lemma 3 is essentially similar to the proof of Lemma 2 and is omitted here due to space limitation.

We can now demonstrate the convergence of the Theorem 1:

Proof of Theorem 5

Replacing $G(v,v_{ab}^{(t)} )$ in Eq. (27) by Eq. (28), we get

$$\begin{aligned} v_{ab}^{(t+1)} =v_{ab}^{(t)} \frac{2(\lambda +1)({\varvec{W}}^{T}{\varvec{K}})_{ab} +2\mu ({\varvec{V}}\mathbf{S })_{ab} }{(2{\varvec{W}}^{T}\varvec{KWV}+\lambda {\varvec{A}}+\lambda {\varvec{B}}+{\varvec{2}}\mu {{\varvec{V}}}{{\varvec{E}}})_{ab} } \end{aligned}$$

Since Eq. (28) is an auxiliary function, $F_{z_{ab} }$ is nonincreasing under this updating rule.

Similarly, Replacing $G(w,w_{ab}^{(t)} )$ in Eq. (27) by Eq. (29), we get

$$\begin{aligned} w_{{ab}}^{(t + 1)} = w_{ab}^{(t)} \frac{\left( {{\varvec{KV}}^{T} + \lambda \sum \nolimits _{i = 1}^{N} {{\varvec{X}}^{T} {\varvec{x}}_{i} {\mathbf {1}}^{T} {\varvec{D}}_{i} } } \right) _{{ab}} }{\left( {{\varvec{KWVV}}^{T} + \lambda \sum \nolimits _{i = 1}^{N} {{\varvec{KWD}}_{i} } } \right) _{{ab}} } \end{aligned}$$

Since Eq. (30) is an auxiliary function, $F_{w_{ab} }$ is nonincreasing under this updating rule.