Manifold NMF with L₂₁ norm for clustering

Abstract

Nonnegative matrix factorization has been widely used in data mining and machine learning fields as a clustering algorithm. The standard nonnegative matrix factorization algorithm utilizes the sum of squared error to measure the quality of factorization, however, the noise and outliers in the dataset will reduce the performance of algorithm significantly. This paper proposes a robust manifold nonnegative matrix factorization algorithm based on L₂₁ norm, and the projected gradient method is utilized to obtain the updating rules. The proposed algorithm utilizes the L₂₁ norm to measure the quality of factorization, which is insensitive to the noise and outliers, also it utilizes the geometrical structure of the dataset and considers the local invariance. The experimental results on several data sets and the comparison with other clustering algorithms demonstrate the effectiveness of the proposed algorithm.

Introduction

The “semantic gap” between image visual features and high-level semantic is the bottleneck of the development of image retrieval. The core problem of improving the utilization efficiency of image content and reducing the semantic gap is how to represent the visual features of the image effectively, that is, how to find a reasonable method to show the data [1], [2]. In order to deal with the data information in the matrix, the matrix is usually decomposed [3]. After the decomposition of the matrix, not only the dimension of the original matrix can be greatly reduced, but also the data stored in the original matrix can be summarized and compressed. The matrix decomposition technology has attracted more and more attention. The typical matrix decomposition methods mainly includes: principal component analysis (PCA) [4], linear discriminant analysis (LDA) [5], independent component analysis (ICA) [6], singular value decomposition (SVD) [7], etc. These methods usually make decomposition or linear transformation on the original data matrix under certain restrictions. They are common in allowing negative values in the decomposition results, which is correct from the view of calculation, but in practice, negative values are meaningless [8].

Recently, non-negative matrix factorization (NMF) as an effective matrix decomposition method has attracted wide attention. In 1999, Lee and Seung [9] first propose the concept of non-negative matrix factorization on ”Nature” journal. Different from the above-mentioned typical matrix decomposition methods, NMF method is unique in that all elements of the matrix decomposition process are non-negative. NMF method decomposes the original matrix into two non-negative matrix: the left matrix is called basis matrix, the right matrix is called coefficient matrix. Therefore, the column vector in the original matrix can be interpreted as the weighted sum of all the columns in the basis matrix, and the weight coefficients are contained in the coefficient matrix. Such a vector combination has an intuitive semantic interpretation, in line with the concept of ”local constitute a whole” in human thinking [10], [11]. The solution methods of NMF include multiplicative update rule, least square substitution method, and projected gradient method [12]. By adding non-negative constraints, NMF can not only ensure the interpretation of the decomposition results, but also has the advantages of easy implementation, small occupation of storage space, and so on. Therefore, it has a certain practical significance to explore the non-negative matrix factorization.

NMF has received considerable attention since it was presented. In order to improve the efficiency and recognition rate of NMF, many scholars have made a deep research on the NMF algorithm, and obtained a lot of research results. Akashi and Okatani [13] introduce sparse encoding theory into NMF. They apply sparseness constraints on both the basis matrix and coefficient matrix, and propose the sparse non-negative matrix factorization algorithm (SNMF). Liu et al. [14] design the constrained non-negative matrix factorization algorithm (CNMF), which uses the known label information to guide the matrix factorization process. Later, Shu and Zhao [15] improve the CNMF by adding sparse constraints, and propose the constrained non-negative matrix factorization with sparseness (CNMFS). Feng et al. [16] propose the weighted non-negative matrix factorization (WNMF), in which each training sample is accompanied by a non-negative weight. Then we may control the influence of the samples on the decomposition results by adjusting the sample weights. Bucak and Gunsel [17] combine incremental learning with non-negative matrix factorization, and propose the incremental non-negative matrix factorization algorithm (INMF), which reduces the computation scale using the idea of block matrix. Kong et al. [18] propose a robust non-negative matrix factorization algorithm based on L₂₁ norm (L₂₁NMF). It overcomes the drawbacks of standard NMF algorithm which is sensitive to the noise and outliers, and improves the robustness of the algorithm. In addition, NMF has been successfully applied to various fields, including text clustering [19], face recognition [20], visual tracking [21], image denoising [22], image retrieval [23], social network analysis [24], etc.

The manifold is a curled space with the properties of Euclidean space in local area. Manifold learning can achieve the dimension reduction by finding the low dimensional manifold embedded in the high-dimensional space. Manifold learning algorithms includes isometric mapping (Isomap), locally linear embedding (LLE), Laplacian eigenmap (LE). They use the local invariant features that two close sample points in high-dimensional space also have short distance in the low dimensional manifold [25]. Cai et al. [26] propose the graph regularized non-negative matrix factorization algorithm (GNMF), which combines manifold learning and NMF. GNMF algorithm considers the geometric structure contained in the original data and will preserve the local geometric information of data sets during the decomposition process. Jiang et al. [27] propose the graph regularized non-negative matrix factorization algorithm with sparse constraints (GNMFSC), which not only takes the geometrical information of data into account, but also impose sparse constraints on the coefficient matrix, so the face images after decomposition have higher recognition rate. But in GNMF algorithm, the quality of matrix decomposition is measured by the Frobenius norm, which is easily affected by the noise and abnormal values in the data. In this case, large errors of the objective function may exist and the performance of the algorithm therefore decrease. In order to get more robust clustering results, this paper presents a manifold non-negative matrix factorization algorithm based on L₂₁ norm (MNMFL₂₁), in which the update rules are calculated by projection gradient method. MNMFL₂₁ algorithm uses the L₂₁ norm to measure the quality of matrix decomposition, so it is not sensitive to the data noise. Also it combines the manifold learning with the robust NMF, and uses the local invariance property to detect the geometric structure of data. The main contribution of this paper are: (1) we use L₂₁ norm to improve the GNMF algorithm and propose MNMFL₂₁ algorithm; (3) we give the objective function of MNMFL₂₁ algorithm and deduce its equivalent form to facilitate the calculation; (4) we present the detailed update rules of the non-negative matrix in MNMFL₂₁ algorithm with the projected gradient method.

The rest parts of the article are as follows: Section 2 introduces the standard NMF algorithm, L₂₁NMF algorithm and GNMF algorithm; Section 3 gives the definition of MNMFL₂₁ algorithm and its equivalence objective function; Section 4 uses the gradient projection method to solve the update rules of MNMFL₂₁ algorithm; Section 5 shows the experimental results of the MNMFL₂₁ algorithm on several data sets and illustrates the validity of MNMFL₂₁ algorithm by comparing it with other clustering algorithms; finally, we summarize the work in this paper.