Elastic nonnegative matrix factorization

Highlights

• ENMF introduces an elastic loss that takes advantage of both Frobenius norm and ℓ_{2, 1} norm when noise distribution is uncertain, therefore ENMF is far more insensitive to noise and outliers.

• ENMF takes the geometric information of the projected data points in the low dimensional manifold as feedback to construct the affinity graph, hence ENMF can handle the situation that a few exceptional data pairs are close in the original space but far away from each other in the manifold.

• ENMF utilizes the exclusive LASSO to enhance the intra-cluster competition and therefore the “winner” is more likely to stand out while the “loser” tends to be out in a sparse manner.

• ENMF provides consistently better clustering results on several well-known data sets as compared to standard NMF and several other variants of the NMF algorithm.

Abstract

Nonnegative matrix factorization (NMF) plays a vital role in data mining and machine learning fields. Standard NMF utilizes the Frobenius norm while robust NMF uses the robust ℓ_2,1-norm to measure the quality of factorization, given the assumption of i.i.d Gaussian noise model and i.i.d Laplacian noise model, respectively. In this paper, we propose a novel elastic loss which is intercalated and adapted between Frobenius norm and ℓ_2,1-norm. Inspired by this, we derive an elastic NMF model guided by the elastic loss with incorporating geometry manifold information while enforcing sparsity of coefficients at intra-cluster level via ℓ_1,2-norm. The new formulation is more robust to noises while preserving the stronger capability of clustering. We propose an EM-like algorithm (using an auxiliary function) to solve the resultant optimization problem, whose convergence can be rigorously proved. The extensive experiments demonstrate the effectiveness of the novel elastic NMF model on benchmarks.

Introduction

In the fields of data mining and machine learning, the actual input data matrix in many applications is often of very high dimension, hence dimensionality reduction is a crucial process before data mining process. The most widely used dimensionality reduction methods include Principle Component Analysis (PCA), Singular Value Decomposition (SVD), Non-negative Matrix Factorization (NMF), etc. Different from PCA and SVD, NMF tends to discover two nonnegative matrices whose product is approximately close to the original matrix. More specifically, given a nonnegative matrix $\mathbf{X}\in {\mathbb{R}}^{p\times n}$ and r ≪ min (p, n), X is approximately factorized into two nonnegative matrices $\mathbf{F}\in {\mathbb{R}}^{p\times r}$ and $\mathbf{G}\in {\mathbb{R}}^{r\times n}$. Thus, the original data point x_i in the p-dimensional space is projected to g_i in a lower k-dimensional subspace defined by columns of F.

Since the initial work by Lee and Seung [1], recent years have seen the expanding research on NMF. NMF has demonstrated its advantages in a variety of areas, such as document clustering [2], [3], multimedia data analysis [4], microarray data analysis [5], social network analysis [6], [7], single channel source separation [8], [9], [10], [11], visual tracking [12], audio source separation [13], [14], detecting topic hierarchies [15], graph matching [16], etc. Algorithmic extensions of NMF also have been extended in order to accommodate a number of data analysis problems, e.g., consensus clustering [17], balanced clustering [18], [19], semi-supervised clustering [20], [21], classification [22], [23], multiple-domain learning [24], multi-kernel learning [25] and collaborative filtering [26].

Standard NMF which utilizes the least square error function to measure the quality of factorization is ideal for zero mean, Gaussian noise. However, in real world cases, many data in various applications may not align with the above situation. It has been proved that the standard NMF is sensitive to outliers [27] which could have significant impact on the objective function with squared residue error. Robust matrix factorization with ℓ₁ norm in [28] reduces the negative impact posed by the noise, but it is unable to preserve the feature rotation invariance which is, however, required by many applications. Robust nonnegative matrix factorization using ℓ_2,1-norm (ℓ_2,1-NMF) in [27] is ideal for the assumption of the i.i.d Laplacian noise model. [29] utilizes a robust capped norm to handle the extreme outliers. Several robust versions of NMF are proposed in [30], including NMF based on the Correntropy Induced Metric (CIM-NMF), row-based CIM-NMF (rCIM-NMF), and NMF based on the Huber function (Huber-NMF).

For many data sets in the area of data mining and machine learning, there is a low dimensional manifold embedded in the high dimensional original space. Manifold learning algorithms including Locally Linear Embedding (LLE) [31], Isometric Mapping (ISOMAP) [32], Laplacian Eigenmap (LE) [33] and Locality Preserving Projections (LPP) [34] aim to detect the underlying manifold structure to improve the learning performance. Several graph based clustering methods [35], [36], [37] have shown the effectiveness by exploiting the locally geometric structure of data. By learning a similarity graph with exactly r connected components (where r equals to the cluster number), the Constrained Laplacian Rank (CLR) [38] and Clustering with Adaptive Neighbors (CAN) [39] methods exhibit excellent performance. Graph Regularized NMF (GNMF) in [40] seeks a matrix factorization which respects the intrinsic geometry of data. Rather than using a fixed graph as GNMF, AdapGrNMF in [41] regularizes NMF with an adaptive graph constructed based on the feature selection results. MultiGrNMF in [42] approximates the intrinsic manifold by a linear combination of several graphs with different models and parameters. The graph dual regularization nonnegative matrix factorization (DNMF) in [43] considers the geometric structures of the data manifold and the feature manifold together. The global discriminative-based nonnegative spectral clustering methods in [44] integrate the geometrical structure and discriminative structure in a joint framework. RMNMF in [45] integrates ℓ_2,1-NMF and spectral clustering with an additional orthogonal constraint. In these algorithms, the geometrical information is usually encoded by an affinity graph, whose vertexes represent the data points and edge weights indicate the affinity between data pairs. Meanwhile, locally invariant idea [46] is considered, i.e., the nearby points in the high dimensional original space are likely close to each other in the low dimensional manifold.

Due to nonnegative constraint, NMF results in sparse basis and coefficient matrices which provide parts-based representation. Such parts-based representation is accordant to the psychological and physiological evidence in the human brain [47], [48], [49]. However, in some cases, the sparsity obtained by NMF is not enough. NMF with sparseness constraints (NMFsc) in [50] enhances the sparsity of the basis and coefficient matrices by explicitly setting both ℓ₁ and ℓ₂ norms. Sparse NMF (SNMF) in [51] imposes the sparsity at intra-data-point level. The global Nonnegative Matrix Underapproximation (G-NMU) and recursive Nonnegative Matrix Underapproximation (R-NMU) methods in [52] utilize an underapproximation technique based on Lagrangian relaxation and provide sparse parts-based representations with low reconstruction error. Logdet divergence based sparse NMF method (LDS-NMF) in [53] deals with the rank-deficiency problem and enhances the sparsity of coefficients using the standard LASSO regularization term.

In this paper, we propose a novel elastic loss which is intercalated and adapted between Frobenius norm and ℓ_2,1-norm. Then we build an elastic NMF model (ENMF) guided by the novel elastic loss. To exploit the locally geometric structure of data, ENMF incorporates a manifold regularization term. However, different from other graph-based algorithms, the affinity graph is constructed based on not only the high dimensional original space but also the low dimensional manifold. Noticing that most data points may conform to the locally invariant idea, there are a few exceptions, i.e., two data points are close to each other in the high dimensional original space, but the distance between them in the low dimensional manifold may be very large. From this perspective, we take the geometric information of the projected data points in the manifold as feedback in order to reduce the edge weights between exceptional data pairs in the affinity graph. What’s more, ENMF incorporates an exclusive LASSO regularization term to enhance the sparsity of coefficients. Noticing that coefficients of each single cluster rather than coefficients of different clusters should compete to survive. As a result, the exclusive LASSO term in the formulation is naturally used to encourage the intra-cluster competition but discourage the inter-cluster competition. Given all the above considerations, we derive the corresponding multiplicative updating algorithm (via an auxiliary function approach) and provide rigorous analysis on its convergence and correctness.

The contributions of our ENMF are summarized as follows.

• ENMF introduces an elastic loss that leverages the advantage of both Frobenius norm and ℓ_2,1 norm when noise distribution is uncertain, therefore ENMF is far more insensitive to noise and outliers.

• ENMF takes the geometric information of the projected data points in the low dimensional manifold as feedback to construct the affinity graph, hence ENMF can handle the situation where a few exceptional data pairs are close in the original space but far away from each other in the manifold.

• ENMF utilizes the exclusive LASSO to enhance the intra-cluster competition and therefore the “winner” is more likely to stand out while the “loser” tends to recede in a sparse manner.

• ENMF provides consistently better clustering results on several well-known data sets as compared to standard NMF and several other variants of the NMF algorithm.

The rest of the paper is organized as follows. In Section 2, we present the formulation of our ENMF and its computational algorithm. In Section 3, we provide a rigorous analysis of the convergence of the algorithm. In Section 4, we prove that the converged solution satisfies the Karush-Kuhn-Tucker (KKT) condition, which further validates the correctness of the algorithm. In Section 5, we show experimental results on several well-known data sets by making comparisons with k-means algorithm, PCA k-means algorithm, standard NMF algorithm and several other variants of the NMF algorithm. Finally, we conclude this paper.