Correntropy induced metric based graph regularized non-negative matrix factorization

doi:10.1016/j.neucom.2015.08.126

Neurocomputing

Volume 204, 5 September 2016, Pages 172-182

https://doi.org/10.1016/j.neucom.2015.08.126 Get rights and content

Abstract

Non-negative matrix factorization (NMF) is a popular dimension reduction method which plays an important role in many pattern recognition and computer vision tasks. However, the low-dimensional representations learned by conventional NMF methods neither taking off the effect of outliers nor preserving the geometric structure in datasets. In this paper, we proposed a correntropy induced metric based graph regularized NMF (CGNMF) to overcome the aforementioned deficiencies. CGNMF maximizes the correntropy between data matrix and its reconstruction to filter out the noises of large magnitudes, and preserves the intrinsic geometric structure of data by using graph regularization. To further enhance the reliability of CGNMF, we proposed correntropy induced metric based graph regularized projective NMF (CGPNMF) to learn clean coefficients by minimizing its distance to the projected samples measured by the correntropy induced metric. Experimental results on popular facial image datasets confirm the effectiveness of both CGNMF and CGPNMF comparing with the state-of-the-arts methods.

Introduction

Dimension reduction makes an important contribution in pattern recognition, subspace selection [23], computer vision [3], [7], [9], [13] and information retrieval [6], [5], [8]. Since dimension reduction reveals the intrinsic structure of data, it can enhance the performance of consequent processing. Among existing dimension reduction methods, non-negative matrix factorization (NMF) [4], [10], [11] has received much attention and become a hot topic in recent years. Particularly, NMF learns two low-dimensional matrices to approximate the original high-dimensional data matrix, meanwhile constrains them to be non-negative. Since NMF can learn a natural parts-based representation, which is consistent with the intuition of learning the parts to form a whole, it has been widely applied to data mining [12], [14], [15], [17], pattern recognition [18], [19], [20], [21], and computer vision [16], [22], [24], [25].

Since the seminal work of Lee and Seung [4], NMF has been continuously improved. For example, Zeferiou et al. [26] proposed discriminant NMF (DNMF) for supervised dimension reduction by incorporating Fisher׳s criterion in NMF. However, DNMF requires the samples obey Gaussian distribution, which is sometimes inconsistent with the assumption of NMF. Cai et al. [27] developed a graph regularized NMF (GNMF) to encode the geometric structure of data by a nearest neighbor (NN) graph. However, traditional NMF methods cannot provide a robust decomposition because their objective function, i.e., L₂-norm based [28] and Kullback–Leibler (KL) divergence based [29] loss functions, are sensitive to outliers. Although Li et al. [33] proposed a robust graph regularized NMF via maximizing correntropy criterion (MCCGR), the constructed adjacent graph is influenced by noisy samples, and thus leads to poor performance on seriously corrupted datasets. From the viewpoint of learning, both robustness of data representation and purity of constructed graph are important in NMF.

In this paper, we propose a correntropy induced metric based graph regularized NMF (CGNMF) to improve the robustness of NMF with the geometric structure of dataset preserved. In particular, we replace the L₂-norm based loss of GNMF with the well-known correntropy induced metric (CIM, [30]) to search a robust matrix decomposition. Since CIM approximates the L₀-norm when the volume of error is large and approximates the L₂-norm when the volume of error is relatively small, it is robust to noise of large magnitudes. CGNMF learns a low-dimensional subspace which preserves the intrinsic geometric structure of dataset via a previously constructed graph from the original data. In addition, to improve the purity of the constructed graph, we improved our CGNMF by constructing the graph with the sparse representation method [31], [32]. To enhance the reliability of CGNMF, we proposed correntropy induced metric based graph regularized projective NMF (CGPNMF) to learn clean coefficients by narrowing its distance to the projected samples measured in the correntropy induced metric sense. Experimental results on popular facial image datasets confirm the effectiveness of both CGNMF and CGPNMF comparing with the state-of-the-arts methods.

The rest of this paper is organized as follows: we briefly reviewed the related NMF variants in Section 2 and presented the correntropy induced metric based graph regularized NMF (CGNMF), its stable version, and the optimization algorithm in Section 3. Section 4 proposed the correntropy induced metric based graph regularized projective NMF (CGPNMF) and its optimization algorithms as well as the proof of convergence. Then we show the experimental results on popular facial image datasets comparing with the representative NMF methods in Section 5. We conclude this paper in Section 6.

Section snippets

Related works

NMF aims to find two non-negative matrices, i.e., $U \in R^{m \times r}$ and $V \in R^{r \times n}$ , to approximate the sample data, i.e., $X \in R^{m \times n}$ , by minimizing the distance between X and UV, where $r ⪡ \min {m, n}$ . Traditional measurement is either squared L₂-norm or Kullback–Leibler (KL)-divergence, and they are not robust enough because their underlying distributions cannot effectively model outliers.

To well preserve the intrinsic geometric structure of the original data, Cai et al. [27] proposed graph regularized NMF (GNNF)

Correntropy induced metric based graph regularized NMF

In this paper, we first presented a correntropy induced metric based graph regularized NMF (CGNMF) based on the correntropy induced metric [30]. Secondly, we improved the reliability of CGNMF by constructing the adjacent graph with sparse representation.

Correntropy induced metric based graph regularized projective NMF

Although CGNMF can learn an effective low dimensional space by inhibiting the influence of outliers to both coefficients and adjacent graph, it is unsatisfactory on some seriously corrupted datasets because the CIM based loss still introduces noisy on the learned coefficients. In this paper, we proposed a correntropy induced metric based graph regularized projective NMF (CGPNMF) by directly inhibiting the effect of outliers to the coefficients. In particular, based on CGNMF, CGPNMF further

Experimental results

In this section, we present several experiments to evaluate the effectiveness of the proposed CGNMF and CGPNMF on both Yale [39] and ORL datasets [40], comparing with GNMF [27], MCCGR [33], RMNMF [34], K-means [41], [42], NMF [28]. Since the initial U and V are selected randomly, we performed 20 independent trials with different number of clusters to compute the average accuracy and the average mutual information. We initialized all algorithms with the same randomly generated U and V, and then

Conclusion

In this paper, we first proposed a correntropy induced metric based graph regularized NMF (CGNMF), and a stable version with sparse representation based adjacent graph construction, and then proposed a correntropy induced metric based graph regularized projective NMF (CGPNMF) by learning clean coefficients. Since CGNMF maximizes the correntropy to denoise the data and constrains the coefficient representation with adjacent graph learned from the original data, it outperforms the other

Acknowledgments

This work was partially supported by the Research Fund for the Doctoral Program of Higher Education of China, SRFDP (under Grant no. 20134307110017) and the Scientific Research Plan Project of NUDT (under Grant no. JC13-06-01 and JC14-06-01) and the National Natural Science Foundation of China (under Grant no. 61502515).

Yuanyuan Wang received both B.S. and M.S. degrees from the National University of Defense Technology, Changsha, China. Now she is a lecturer with the Army Officer Academy. His current research interests include computer vision and machine learning.

References (42)

W.F. Liu et al.
Multiview Hessian discriminative sparse coding for image annotation
Comput. Vis. Image Underst.
(2014)
B.D. Liu et al.
Learning dictionary on manifolds for image classification
Pattern Recognit.
(2013)
B. Mao, N.Y. Guan, D.C. Tao, X.H. Huang, Z.G. Luo, Correntropy induced metric based graph regularized non-negative...
W.J. Zhang, N.Y. Guan, D.C. Tao, B. Mao, X.H. Huang, Z.G. Luo, Correntropy supervised non-negative matrix...
D.C. Tao et al.
General tensor discriminant analysis and Gabor features for gait recognition
IEEE Trans. Pattern Anal. Mach. Intell.
(2007)
D.D. Lee et al.
Learning the parts of objects by non-negative matrix factorization
Nature
(1999)
J. Yu et al.
Learning to rank using user clicks and visual features for image retrieval
IEEE Trans. Cybern.
(2015)
D.C. Tao et al.
Asymmetric bagging and random subspace for support vector machines based relevance feedback in image retrieval
IEEE Trans. Pattern Anal. Mach. Intell.
(2006)
C. Xu et al.
Multiview intact space learning
IEEE Trans. Pattern Anal. Mach. Intell.
(2015)
J. Yu et al.
Click prediction for web image reranking using multimodal sparse coding
IEEE Trans. Image Process.
(2014)

J. Yu et al.

High-order distance based multiview stochastic learning in image classification

IEEE Trans. Cybern.

(2014)

N.Y. Guan et al.

Nonnegative patch alignment framework

IEEE Trans. Neural Netw.

(2011)

N.Y. Guan et al.

Manifold regularized discriminative nonnegative matrix factorization with fast gradient descent

IEEE Trans. Image Process.

(2011)

C. Liu, H.C. Yang, J.L. Fan, L.W. He, Y.M. Wang, Distributed nonnegative matrix factorization for web-scale dyadic data...

J. Yu et al.

Adaptive hypergraph learning and its applications in image classification

IEEE Trans. Image Process.

(2012)

B. Geng et al.

Ensemble manifold regularization

IEEE Trans. Pattern Anal. Mach. Intell.

(2012)

N.Y. Guan et al.

NeNMFAn optimal gradient method for nonnegative matrix factorization

IEEE Trans. Signal Process.

(2012)

J. Yu, D.C. Tao, Modern Machine Learning Techniques and Their Applications in Cartoon Animation Research, Piscataway,...

N.Y. Guan et al.

Online nonnegative matrix factorization with robust stochastic approximation

IEEE Trans. Neural Netw. Learn. Syst.

(2012)

S.Z. Li, X.W. Hou, H.J. Zhang, et al., Learning spatially localized, parts-based representation, in: Proceedings of the...

A.D. Pascual-Montano et al.

Nonsmooth nonnegative matrix factorization (nsNMF)

IEEE Trans. Pattern Anal. Mach. Intell.

(2006)

Cited by (21)

Inter- and intra-hypergraph regularized nonnegative matrix factorization with hybrid constraints
2024, Engineering Applications of Artificial Intelligence
The accurate low rank representation of high-dimensional data learned by the manifold regularized nonnegative matrix factorization framework is effective in data clustering. In previous work, this work has mainly been solved in the way of similarity matrix induction. To further increase the efficacy of low-rank representations, we propose a novel semi-supervised non-negative matrix factorization (NMF) model in this study called inter- and intra-hypergraph regularized non-negative matrix factorization with hybrid constraints (IGNMFC). IGNMFC constructs intra-hypergraph regularization and intra-hypergraph regularization by hypergraph learning, which can precisely induce high-dimensional data to map toward low-dimensional. Moreover, hybrid constraints are introduced to improve the exclusivity and sparsity of low-dimensional representations, and the result accounts for the benefit of this method in learning distinguishable subspace representations. Finally, IGNMFC is transformed into an optimal problem and an efficient iteration rule is proposed. Experiments on several datasets demonstrate that the proposed method outperforms the state-of-the-art NMF algorithms, and can achieve at least 7.9% $\sim$ 15% accuracy improvement in most cases.
Noise-tolerant clustering via joint doubly stochastic matrix regularization and dual sparse coding
2023, Expert Systems with Applications
Clustering has received a lot of attention and research in many important fields, such as machine learning and data mining. Especially, the clustering method based on non-negative matrix factorization (NMF) has been widely used. However, the following problems still exist. First, the clustering method based on the traditional NMF cannot handle noise and outliers well, although the reconstruction error can be measured by the $l_{2, 1}$ -norm instead Frobenius norm to improve the robustness, the effect is not obvious. Second, NMF based on graph regularization mostly relies on initial similarity graph, the method of constructing the graph is fixed and cannot update the graph adaptively. Third, there is no sparse constraints both on the basis matrix and the coefficient matrix, which may cause important information to be ignored during clustering. Therefore, in order to solve the above problems, we propose a joint doubly stochastic matrix regularization and dual sparse coding framework (DSNMF). Specifically, we use correntropy instead of the Euclidean distance to overcome the influence of non-Gaussian noise and outliers, making NMF more robust. In addition, the adaptive graph learning can learn a high-quality graph through doubly stochastic matrix to fully maintain local smoothness. Furthermore, we perform sparse coding both on the basis matrix and the coefficient matrix to make full use of the sparsity of the matrix. Finally, the experimental results on eleven datasets show that our method is better than other methods in most cases, and our DSNMF is robust to noise and outliers.
Noise-aware clustering based on maximum correntropy criterion and adaptive graph regularization
2023, Information Sciences
Graph-based clustering is a basic subject in the field of machine learning, but most of them still have the following deficiencies. First, similarity graph construction and data division into corresponding classes are always divided into two independent steps. Second, noise contained in real data may cause the learned similarity graph to be inaccurate. Third, the traditional metrics based on Euclidean distance is difficult to tackle non-Gaussian noise. In order to eliminate these limitations, a noise-aware clustering based on correntropy and adaptive graph regularization method (NCCAGR) is proposed. 1) In order to change the problem from two-steps to single-step, we formulate a joint clustering learning framework that simultaneously learns a robust similarity graph and performs data clustering; 2) To overcome the influence of noise, we construct a Laplacian matrix and perform adaptive graph regularization based on clean data; 3) By introducing the correntropy to solve the problem of non-Gaussian noise and heavy tail in the original data. Furthermore, a half-quadratic optimization method is used to transform the problem into a quadratic form to facilitate subsequent solutions. Finally, experiments show that the proposed method not only has high performance, but also outperforms both classical methods and state-of-the-art methods in robustness.
Robust semi-supervised nonnegative matrix factorization for image clustering
2021, Pattern Recognition
Citation Excerpt :
For example, the graph regularized NMF methods have been proposed by using a nearest neighborhood graph to explore the intrinsic geometrical structure of data [17–20]. In order to suppress the negative influence of non-Gaussian noise with complicated statistical distribution in the data, several robust loss functions such as l2,p-norm and correntropy have been applied in NMF for developing some robust NMF methods [21–24]. Up to now, most of the existing NMF approaches are unsupervised, and pay little attention to the supervised information hidden in the data.
Nonnegative matrix factorization (NMF) is a powerful dimension reduction method, and has received increasing attention in various practical applications. However, most traditional NMF based algorithms are sensitive to noisy data, or fail to fully utilize the limited supervised information. In this paper, a novel robust semi-supervised NMF method, namely correntropy based semi-supervised NMF (CSNMF), is proposed to solve these issues. Specifically, CSNMF adopts a correntropy based loss function instead of the squared Euclidean distance (SED) in constrained NMF to suppress the influence of non-Gaussian noise or outliers contaminated in real world data, and simultaneously uses two types of supervised information, i.e., the pointwise and pairwise constraints, to obtain the discriminative data representation. The proposed method is analyzed in terms of convergence, robustness and computational complexity. The relationships between CSNMF and several previous NMF based methods are also discussed. Extensive experimental results show the effectiveness and robustness of CSNMF in image clustering tasks, compared with several state-of-the-art methods.
Robust non-negative matrix factorization with multiple correntropy-induced hypergraph regularizer
2020, Signal Processing
Citation Excerpt :
Our method is closely relevant to correntropy induced NMF. Wang et al. [14] Proposed a Correntropy induced graph egularized NMF method, where the correntropy induced metric is used to measure the reconstruction errors and the simple graph Laplacian regularizer is used to preserve the intrinsic manifold structure. Zhang et al. [15] proposed Correntropy Supervised NMF, where not only the correntropy metrix is used, but also the data label information is leveraged to employ the classification task.
Non-negative matrix factorization (NMF) is a popular learning tool, which has widely used in computer vision and image processing. Many variants and extensions of NMF have been proposed, where the manifold regularized NMF methods have achieved promising performance due to the preservation of the geometric structures of the data. However, for many applications, the data is usually contaminated by complex noise. The noise leads the data to deviate from the intrinsic manifold, resulting in the degenerate performance. To make the NMF methods reflect the underlying manifold structure well, we propose a novel robust non-negative matrix factorization model. Our model proposes a novel ensemble manifold regularizer, which combines multiple robust hypergraphs to estimate the underlying manifold. Specifically, the correntropy-induced hypergraph is used as the initial manifold estimation. By incorporating the proposed regularizer into the original NMF framework, two novel manifold regularized NMF methods are proposed. The clustering results on the noisy image datasets demonstrate that our model is effective, which achieves the state-of-the-art performance.
Robust orthogonal nonnegative matrix tri-factorization for data representation
2020, Knowledge-Based Systems
Nonnegative matrix factorization (NMF) has been a vital data representation technique, and has demonstrated significant potential in the field of machine learning and data mining. Nonnegative matrix tri-factorization (NMTF) is an extension of NMF, and provides more degrees of freedom than NMF. In this paper, we propose the correntropy based orthogonal nonnegative matrix tri-factorization (CNMTF) algorithm, which is robust to noisy data contaminated by non-Gaussian noise and outliers. Different from previous NMF algorithms, CNMTF firstly applies correntropy to NMTF to measure the similarity, and preserves double orthogonality conditions and dual graph regularization. We adopt the half-quadratic technique to solve the optimization problem of CNMTF, and derive the multiplicative update rules. The complexity issue of CNMTF is also presented. Furthermore, the robustness of the proposed algorithm is analyzed, and the relationships between CNMTF and several previous NMF based methods are discussed. Experimental results demonstrate that the proposed CNMTF method has better performance on real world image and text datasets for clustering tasks, compared with several state-of-the-art methods.

View all citing articles on Scopus

Shuyi Wu received the B.S. degree from the National University of Defense Technology, Changsha, China, where he is currently working toward the M.S. degree with the School of Computer Science. His current research interests include computer vision and image processing.

Bin Mao received the both B.S. and M.S. degrees from the National University of Defense Technology, Changsha, China, where he is currently working toward the Ph.D. degree with the School of Computer Science. His current research interests include machine learning and data mining.

Xiang Zhang received the B.S. and M.S. degrees from the Anhui University and National University of Defense Technology, respectively. He is currently working toward his Ph.D. degree with the School of Computer Science. His current research interests include computer vision and image processing.

Zhigang Luo received the B.S., M.S., and Ph.D. degrees from the National University of Defense Technology, Changsha, China, where he is currently a Professor with the School of Computer Science. His current research interests include artificial intelligence and machine learning.

View full text

Correntropy induced metric based graph regularized non-negative matrix factorization

Abstract

Introduction

Section snippets

Related works

Correntropy induced metric based graph regularized NMF

Correntropy induced metric based graph regularized projective NMF

Experimental results

Conclusion

Acknowledgments

Comput. Vis. Image Underst.

Pattern Recognit.

General tensor discriminant analysis and Gabor features for gait recognition

IEEE Trans. Pattern Anal. Mach. Intell.

Learning the parts of objects by non-negative matrix factorization

Nature

Learning to rank using user clicks and visual features for image retrieval

IEEE Trans. Cybern.

Asymmetric bagging and random subspace for support vector machines based relevance feedback in image retrieval

IEEE Trans. Pattern Anal. Mach. Intell.

Multiview intact space learning

IEEE Trans. Pattern Anal. Mach. Intell.

Click prediction for web image reranking using multimodal sparse coding

IEEE Trans. Image Process.

High-order distance based multiview stochastic learning in image classification

IEEE Trans. Cybern.

Nonnegative patch alignment framework

IEEE Trans. Neural Netw.

Manifold regularized discriminative nonnegative matrix factorization with fast gradient descent

IEEE Trans. Image Process.

Adaptive hypergraph learning and its applications in image classification

IEEE Trans. Image Process.

Ensemble manifold regularization

IEEE Trans. Pattern Anal. Mach. Intell.

NeNMFAn optimal gradient method for nonnegative matrix factorization

IEEE Trans. Signal Process.

Online nonnegative matrix factorization with robust stochastic approximation

IEEE Trans. Neural Netw. Learn. Syst.

Nonsmooth nonnegative matrix factorization (nsNMF)

IEEE Trans. Pattern Anal. Mach. Intell.