Sparse dual graph-regularized NMF for image co-clustering

Abstract

Nonnegative matrix factorization (NMF) as fundamental technique for clustering has been receiving more and more attention. This is because it can effectively reduce high dimensional data and produce parts-based, linear image representations of nonnegative data. For practical clustering tasks, NMF ignores the geometric structures of both data manifold and feature manifold. In addition, recent research results showed that leveraging sparseness can greatly improve the ability of learning parts. Motivated by the two aspects above mentioned, we propose a novel co-clustering algorithm to enhance the clustering performance, called sparse dual graph-regularized nonnegative matrix factorization (SDGNMF). It aims for finding a parts-based, linear representation of the non-negative data and facilitating the learning tasks. SDGNMF jointly incorporates the dual graph-regularized and sparseness constraints as additional conditions to uncover the intrinsic geometrical, discriminative structures of the data space and feature space. The iterative updating scheme for the optimization problem of SDGNMF and its convergence proofs are also given in detail. Experimental results of clustering on three benchmark datasets demonstrated that SDGNMF algorithm outperforms the compared state-of-the-art methods in image co-clustering.

Introduction

With the increasing development of modern digital technologies for the Big Data Era, a huge amount of image data emerges every moment in a broad range of real-word applications, e.g., machine learning [1], [2] and computer vision [3], [4], [5]. How to retrieve, classify and extract the valuable information from the massive image data efficiently has become a huge challenge for many scholars. Clustering technologies have received considerable attention as a very useful tool to handle a vast number of image data with successful applications in data analysis tasks. Generally speaking, clustering technologies have three problems to be solved for high-dimensional image data: (1) dimensionality reduction; (2) linear representations; (3) better clustering performance. As an unsupervised learning mechanism, clustering seeks the appropriate partitioning of the unlabeled data points with their similarities, which is evaluated by specific metrics.

Due to the efficiency and fast convergence, clustering has been widely studied for many years, and a large number of clustering algorithms have been proposed up till the present moment. Clustering can be divided into constraint-based methods [6], [7] and metric-based [8], [9] methods. Some typical representatives are K-means [10], [11], spectral clustering [12], [13] and non-negative matrix factorization (NMF) [14], [15]. Among the canonical clustering techniques, NMF as a novel clustering method that emerged in recent years has received much more attention from academia and industry. NMF aims to find two low-rank nonnegative matrices, which can provide a better approximation to the original nonnegative matrix and lead to a parts-based representation of the decomposition results for multi-type data. As a result, NMF algorithm has been successfully applied in computer vision, data mining and other fields.

NMF provides numerous problem formulation techniques and algorithmic methods for clustering problems, especially constraint-based clustering. But, it may not guaranteed that better clustering performance can be achieved using the original NMF method, which only requires factorized matrix nonnegative. Therefore, some NMF variants were presented for the requirements of different clustering performance, which imposed other constraints besides non-negativity. Hoyer [16] presented a sparse non-negative matrix factorization (SNMF) algorithm, in which sparseness constraint is enforced as a penalty term of the NMF framework to enhance the ability of learning parts. Cai et al. [17] proposed a graph-regularized non-negative matrix factorization (GNMF) approach to encode the geometrical information of the data space, which preserves the intrinsic geometric structure and simultaneously obtains a compact representation of the hidden semantic information. Liu et al. [18] introduced a constrained non-negative matrix factorization (CNMF) method, which considers a few hard priori label information as additional constraints and integrates it into NMF model to improve the discrimination ability.

However, the algorithms mentioned above consider only one-side clustering, which groups similar objects. Clustering the duality between data points and features, several co-clustering [19], [20], [21], [22] algorithms have been proposed to solve the one-side clustering problems and shown to be superior to traditional one-side clustering. Gu et al. [23] proposed a dual regularized co-clustering (DRCC) based on semi-nonnegative matrix tri-factorization, which constructs two graphs, i.e., data graph and feature graph, to explore the geometric structure of data manifold and feature manifold. Experiments of clustering on many benchmark datasets showed that DRCC outperforms many state-of-the-art clustering methods. Shang et al. [24] presented a novel co-clustering algorithm called graph dual regularization non-negative matrix factorization (DNMF), which simultaneously considers the geometric structures of both the data manifold and the feature manifold. Moreover, they also present a graph dual regularization non-negative matrix tri-factorization (DNMTF) as an extension of DNMF. Experimental results on multi-type datasets demonstrated the effectiveness of both DNMF and DNMTF.

Co-clustering can categorize data points as well as features simultaneously and solve the inefficient problems of one-side clustering, which has an extensive range of application, e.g., text mining [25], recommendation systems [26], [27], gene expression [28], [29] and so on. Yet the existing NMF-based co-clustering methods [30], [31], [32] are very successful in both dimensionality reduction and better clustering performance, it does not always result well in parts-based representations of the image data. The above-mentioned co-clustering algorithms do not simultaneously consider the valuable information of image data, such as the geometric structure of data manifold and feature manifold, the label information, sparseness, etc., which not only deteriorates performance and robustness but also fails to get a better parts-based, linear representation for image clustering. This greatly limits the development of co-clustering algorithms in various fields. In order to solve this problem, while further improving the clustering performance of NMF, we propose a novel algorithm called sparse dual graph-regularized nonnegative matrix factorization (SDGNMF) for image co-clustering, which not only preserves the geometric structures of data manifold and feature manifold by integrating the prior label information into two graphs, but also introduces the Frobenius norm sparsity constraint on the basis matrix factor in NMF objective function to enhance the ability of learning parts. What is more, we construct the objective function of SDGNMF in the form of the Euclidean distance for convenience, the iterative updating scheme is chosen to optimize it and the convergence proofs of the raised method are also given. Finally, we carried out the extensive experiments on three benchmarks to validate the effectiveness and efficiency of the novel co-clustering method proposed in this paper.

The main contributions of our work can be summarized as follows:

• The proposed method possesses the merits of DNMF, which also introduces the prior label information into two graphs to encode the intrinsic geometrical and discriminative structures of the data space as well as feature space, and also takes the label information as additional constraints to decompose matrix. Since the algorithm presented here incorporates the virtues of the two facets mentioned above, it can have better performance on clustering accuracy and normalized mutual information.

• Recent research results showed that the sparsity of the factors can greatly affect the discrimination ability of the learning parts, and if the sparseness levels of the factors are improved, the discrimination ability will also be increased accordingly. Therefore, we exploited sparseness constraints as an independent penalty term to make the basis vectors much sparser. This scheme will raise the sparsity of data that performs other learning tasks like clustering and thus learn a better parts-based, linear representation of image data.

• To the best of our knowledge, this work is the first to integrate the prior label information, sparseness constraints and intrinsic geometric structures in both data manifold and feature manifold into a unified optimization framework. This greatly improves the clustering performance of NMF while reducing the dimension of image data, and obtains a parts-based, linear representation so that further learning tasks can be facilitated.

The remainder of this paper is arranged as follows. First, we present a brief overview of some related methods, including NMF, SNMF and DNMF in Section 2. Next, our approach is described in details and the efficient iterative updating method with proved convergence is developed as well in Section 3. Then Section 4 reports the experimental results and the corresponding analyses on three public datasets. Finally, this paper is concluded in Section 5.