Adaptive local learning regularized nonnegative matrix factorization for data clustering

Abstract

Data clustering aims to group the input data instances into certain clusters according to the high similarity to each other, and it could be regarded as a fundamental and essential immediate or intermediate task that appears in areas of machine learning, pattern recognition, and information retrieval. Clustering algorithms based on graph regularized extensions have accumulated much interest for a couple of decades, and the performance of this category of approaches is largely determined by the data similarity matrix, which is usually calculated by the predefined model with carefully tuned parameters combination. However, they may lack a more flexible ability and not be optimal in practice. In this paper, we consider both discriminative information as well as the data manifold in a matrix factorization point of view, and propose an adaptive local learning regularized nonnegative matrix factorization (ALLRNMF) approach for data clustering, which assumes that similar instance pairs with a smaller distance should have a larger probability to be assigned to the probabilistic neighbors. ALLRNMF simultaneously learns the data similarity matrix under the assumption and performs the nonnegative matrix factorization. The constraint of the similarity matrix encodes both the discriminative information as well as the learned adaptive local structure and benefits the data clustering on manifold. In order to solve the optimization problem of our approach, an effective alternative optimization algorithm is proposed such that our objective function could be decomposed into several subproblems that each has an optimal solution, and its convergence is theoretically guaranteed. Experiments on real-world benchmark datasets demonstrate the superior performance of our approach against the existing clustering approaches.

Appendices

Appendix A: Proof of Theorem 2

Proof

We rewrite (33) as

$$\begin{array}{@{}rcl@{}} L(\textbf{U})=tr(-2\textbf{V}\textbf{X}^{T}\textbf{U} +\textbf{U}\textbf{V}\textbf{V}^{T}\textbf{U}^{T}). \end{array} $$

(39)

By applying Lemma 2, we have

$$\begin{array}{@{}rcl@{}} tr(\textbf{U}\textbf{V}\textbf{V}^{T}\textbf{U}^{T}) \leq\sum\limits_{ij}\frac{(\textbf{U}^{\prime}\textbf{V}\textbf{V}^{T})_{ij}\textbf{U}^{2}_{ij}} {\textbf{U}^{\prime}_{ij}} . \end{array} $$

To obatin the lower bound for the remaining terms, we use the inequality that

$$\begin{array}{@{}rcl@{}} z\geq 1+log z, \forall z\geq 0. \end{array} $$

(40)

Then

$$\begin{array}{@{}rcl@{}} tr(\textbf{V}\textbf{X}^{T}\textbf{U})\geq \sum\limits_{ij}(\textbf{X}\textbf{V}^{T})_{ij}\textbf{U}^{\prime}_{ij}\left( 1+log\frac{\textbf{U}_{ij}}{\textbf{U}_{ij}^{\prime}}\right) . \end{array} $$

By summing over all the bounds, we can get g(U, U^′), which obviously satisfies: (1) g(U, U^′) ≥ J_ALLRNMF(U); (2) g(U, U) = J_ALLRNMF(U).

To find the minimum of g(U, U^′), we take the Hessian matrix of g(U, U^′)

$$\begin{array}{@{}rcl@{}} \frac{\partial^{2}g(\textbf{U}, \textbf{U}^{\prime})}{\partial \textbf{U}_{ij}\partial \textbf{U}_{kl}}=\delta_{ik} \delta_{jl}\left( \frac{2(\textbf{U}^{\prime}\textbf{V}\textbf{V}^{T})_{ij}}{\textbf{U}^{\prime}_{ij}} + 2(\textbf{X}\textbf{V}^{T})_{ij}\frac{\textbf{U}^{\prime}_{ij}}{\textbf{U}^{2}_{ij}}\right) \end{array} $$

which is a diagonal matrix with positive diagonal elements. So g(U, U^′) is a convex function of U, and we can obtain the global minimum of g(U, U^′) by setting $\frac {\partial g(\textbf {U}, \textbf {U}^{\prime })}{\partial \textbf {U}_{ij}}= 0$ and solving for U, from which we can get (34). □

Appendix B: Proof of Theorem 4

Proof

We rewrite (35) as

$$\begin{array}{@{}rcl@{}} L(\textbf{V})\!&=&\!tr\left( -2\textbf{X}^{T}\textbf{U}\textbf{V} +\textbf{V}^{T}\textbf{U}^{T}\textbf{U}\textbf{V}\right.\\ &&\qquad\left.-\lambda \textbf{V}\textbf{L}_{S}^{+}\textbf{V}^{T} +\lambda \textbf{V}\textbf{L}_{S}^{-}\textbf{V}^{T}\right). \end{array} $$

(41)

By applying Lemma 2, we have

$$\begin{array}{@{}rcl@{}} tr(\textbf{V}^{T}\textbf{U}^{T}\textbf{U}\textbf{V})&\leq&\sum\limits_{ij} \frac{(\textbf{U}^{T}\textbf{U}\textbf{V}^{\prime})_{ij}\textbf{V}^{2}_{ij}}{\textbf{V}^{\prime}_{ij}},\\ tr(\textbf{V}\textbf{L}^{-}_{S}\textbf{V}^{T})&\leq&\sum\limits_{ij} \frac{(\textbf{V}^{\prime}\textbf{L}_{S}^{-})_{ij}\textbf{V}^{2}_{ij}}{\textbf{V}^{\prime}_{ij}}. \end{array} $$

To obatin the lower bound for the remaining terms, we use the inequality in (40), then

$$\begin{array}{@{}rcl@{}} tr(\textbf{X}^{T}\textbf{U}\textbf{V})&\geq& \sum\limits_{ij}(\textbf{U}^{T}\textbf{X})_{ij}\textbf{V}^{\prime}_{ij} \left( 1+log\frac{\textbf{V}_{ij}}{\textbf{V}_{ij}^{\prime}}\right),\\ tr(\textbf{V}\textbf{L}^{+}_{S}\textbf{V}^{T})&\geq& \sum\limits_{ijk}(\textbf{L}^{+}_{S})_{jk}\textbf{V}_{ij}^{\prime}\textbf{V}^{\prime}_{ik}\left( 1+log\frac{\textbf{V}_{ij} \textbf{V}_{ik}} {\textbf{V}_{ij}^{\prime}\textbf{V}_{ik}^{\prime}}\right), \end{array} $$

By summing over all the bounds, we can get g(V, V^′), which obviously satisfies: (1) g(V, V^′) ≥ J_ALLRNMF(V); (2) g(V, V) = J_ALLRNMF(V).

To find the minimum of g(V, V^′), we take the Hessian matrix of g(V, V^′)

$$\begin{array}{@{}rcl@{}} \frac{\partial^{2}g(\textbf{V}, \textbf{V}^{\prime})}{\partial \textbf{V}_{ij}\partial \textbf{V}_{kl}}&=&\delta_{ik} \delta_{jl} \left( \frac{2(\textbf{U}^{T}\textbf{X}+ 2\lambda\textbf{L}^{+}_{S})_{ij}\textbf{V}^{\prime}_{ij}}{\textbf{V}^{2}_{ij}}\right.\\ &&\qquad\qquad\left.+\frac{2(\textbf{U}^{T}\textbf{U}\textbf{V}^{\prime}+\lambda\textbf{V}^{\prime}\textbf{L}^{-}_{S})_{ij}}{\textbf{V}^{\prime}_{ij}}\right) \end{array} $$

which is a diagonal matrix with positive diagonal elements. So g(V, V^′) is a convex function of V, and we can obtain the global minimum of g(V, V^′) by setting $\frac {\partial g(\textbf {V}, \textbf {V}^{\prime })}{\partial \textbf {V}_{ij}}= 0$ and solving for V, from which we can get (36). □