Elsevier

Neurocomputing

Volume 487, 28 May 2022, Pages 75-85
Neurocomputing

Multi-view unsupervised feature selection with tensor low-rank minimization

https://doi.org/10.1016/j.neucom.2022.02.005Get rights and content

Abstract

To describe objects more comprehensively and accurately, multi-view learning has attracted considerable attention. Recently, graph embedding based multi-view feature selection methods have been proposed and shown efficient in many real applications. The existing methods generally construct one common graph matrix to exploit the local structure of multi-view data via the linear weight fusion or learning one common graph matrix across all views. However, since all views share the identical graph structure, this emphasizes the consistency too much, resulting in restricting the diversity among different views. In this paper, a tensor low-rank constrained graph embedding method is proposed for multi-view unsupervised feature selection. To embody the view-specific information of each view, our model constructs the graph structure for each corresponding view, respectively. To capture the consistency across views, a tensor low-rank regularization constraint is imposed on the tensor data formed by these graph matrices. An efficient optimization algorithm with theoretical convergence guarantee is designed to solve the proposed method. Extensive experimental results validate that the proposed method outperforms some state-of-the-art methods. The code of our model can be found athttps://www.researchgate.net/publication/353902948_demoTLR.

Introduction

In many applications, the objects are often represented by different modalities or features and each of them has specific physical meaning and statistic property. For example, the remote sensing technique can use various sensors to detect a target such as the image features and the electromagnetic wave features [32]. Different visual descriptors such as SIFT [31], HOG [10] and GIST [29] can be used to represent images. Although different views reflect the same object, the feature spaces could be different among multiple views, which is known as heterogeneous features. Traditional single-view methods can only handle multi-view data by simply concatenating multi-view features as a new single feature set. However, the drawback is that it may ignore the correlation information among multiple views, which may degrade the performance or fail to work. In order to acquire more comprehensive information of an object from multi-view data, it is necessary to discover the complementary information provided by different views and the inner consistency information among all views. To process this issue, many multi-views methods [40], [47], [27] have been proposed and made a contribution to many applications such as bioinformatics, hyperspectral remote sensing, and data recovery [5], [41], [20], [38], [35]. To integrate the multi-view data for clustering, Chen et al. [6] propose to learn a global structure and cluster the multi-view data in embedding space, while Kang et al. [21] propose to utilize the multi-view information by fusing partitions. By leveraging the consistency and diversity simultaneously, Huang et al. [18] propose a unified framework for structured multi-view clustering. Considering the effect of cluster size, Peng et al. [30] propose a multi-view clustering method without parameter selection on cluster size, which aims to learn a space with geometric consistency and cluster assignment consistency. To explore the nonlinear relationships and consider the imbalance among different views, Huang et al. [17] propose an auto-weighted multi-view clustering method that learns multi-view similarity relationships in kernel spaces. For solving the view-insufficiency issue, Huang et al. [16] propose to simultaneously recover the latent intact space from multiple insufficient views and discover the cluster structure from the intact space.

In the field of multi-view learning, multi-view data also brings the problem of a surge in the number of features including noise and redundancy. Multi-view feature selection methods, aiming to reduce the dimensionality of multiple views data and explore the correlation between views, have arisen considerable research interests in recent years [36], [11], [15]. Traditional graph-based multi-view unsupervised feature selection methods construct one common graph matrix to exploit the local structure of multi-view data. Currently, there exist two ways to construct the common graph structure as follows. One is to fuse multiple graphs into one common graph by linear weighted fusion, i.e., S=v=1VαvS(v), where αv is the weight coefficient of v-th view and S(v) is the graph matrix of v-th view. However, in this way, the graph structure needs to be computed in advance and fixed during the embedding procedure. Hence, these pre-defined graphs are inappropriate to exploit the local structure of the multi-view features in the embedding space. The other way is to learn a common graph structure across all views, which assumes that all views share the identical graph structure S. Nevertheless, the aim of these two ways is to yield one common graph matrix to represent the local structure of muti-view data, which emphasizes consistency too much while the diversities among different views are restricted, i.e., the strong consistency problem.

In this paper, we propose a tensor low-rank constrained graph embedding method for multi-view unsupervised feature selection, which considers both consistency and diversity compatible well. In Fig. 1, one can see that the strong consistency constraint makes all views share the same graph. If these graphs {S,S,,S} are stacked into a 3-order tensor N×N×V, we find that the rank of its lateral slice (:,i,:) is equal to 1. First of all, lateral slide is V×N sizes and it embodies the correlations among different views and samples. Secondly, based on the observation, the rank of lateral slide is 1. It means that each view’s graph provides the same information without any other complementary information, i.e., diversity. To capture the consistency and complementary information of multi-view data, we impose a low-rank constraint on lateral slides. In other words, by introducing a low-rank constraint on lateral slice instead of limiting its rank to 1, we can preserve the consistent information between views and exploit the diversity information at the same time. Therefore, we consider to respectively compute the graph matrix for its corresponding view and adopt a tensor low-rank constraint on this graph tensor formed by different graph matrices. On the one hand, our method can obtain the different graph structures for different views to preserve the diversity information. On the other hand, the tensor low-rank constraint on graph tensor can explore the high-order consistency of multiple views. Hence, our work makes consistency and diversity compatible well. Our main contributions are summarized as follows:

  • (1) To exploit the complementary information, we consider computing respectively the graph matrix for each view in the embedding space. To capture the latent consistency across views, we impose a tensor low-rank constraint on a tensor, which is stacked by these learned graph matrices. Both diversity and consistency information can be well ensured in our proposed model.

  • (2) An effective algorithm is presented to solve the optimization problem, together with the theoretical analyses on its convergence and computational complexity. Experimental results on several multi-view databases demonstrate the effectiveness of our proposed method and surpass some state-of-the-art competitive methods.

The paper is organized as follows. In Section 2, we briefly review and discuss the related works. In Section 3, we propose a tensor low-rank constrained graph embedding method for multi-view unsupervised feature selection, together with the theoretical analysis of convergence and computational complexity. Experiments on several multi-view data sets are conducted in Section 4. Section 5 concludes the paper.

Section snippets

Related works

Unsupervised data dimensionality reduction techniques have arisen lots of research interests and can be divided into two categories, i.e., subspace learning [19], [44] and feature selection [34]. In this paper, we focus on the unsupervised feature selection methods, which are approximately divided into three groups: wrapper, filter, and embedded. Wrapper methods select the optimal feature set iteratively with respect to a predetermined feature evaluated function. The typical wrapper method is

Formulation

We propose a multi-view unsupervised feature selection model by using graph embedding technique. In our model, we construct the different graph structures for multiple views in the embedding space. Besides, a tensor low-rank constraint is imposed on the tensor data, which is stacked by these graph matrices, to capture the high-order consistency across views. Hence, our model is formulated as follow:argminW(v),S(v)v=1Vi=1nj=1nW(v)Txi(v)-W(v)Txj(v)2Sij(v)+λ1v=1VW(v)2,1+λ2Ss.t.W(v)TW(v)=

Data sets and experiment setup

Before conducting experiments, we first give the details of six databases in Table 1. We employ several competing unsupervised feature selection methods to evaluate the performance of our method, including PCAScore [2], SPEC [48], LapScor [14], NDFS [24], WMCFS [42], AUMFS [11], ASVW [15], CRV-DCL [33]. To examine the selected features are effective or not, we use the clustering result of all features as the baseline. In the clustering experiments, unsupervised feature selection methods give

Conclusion

This paper proposes a tensor low-rank constrained graph embedding model for multi-view unsupervised feature selection, named TLR-MUFS. By adaptively learning the graphs under tensor low-rank constraints, our proposed method can solve the problem of keeping the view-specific information and consistency information in the multi-view graphs simultaneously. The view-specific information is retained in the graph corresponding to each view, while the consistency information is captured by the tensor

CRediT authorship contribution statement

Haoliang Yuan: Conceptualization, Methodology, Formal analysis, Writing - original draft. Junyu Li: Data curation, Software, Validation, Writing - review & editing. Yong Liang: Supervision, Funding acquisition, Writing - review & editing. Yuan Yan Tang: Supervision, Funding acquisition, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This research work was supported in part by a grant from the “Macao Young Scholars Program” (Project code: AM201915), in part by the National Nature Science Foundation of China under Grant 61903091 and 62172458, in part by Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010801), and in part by the Science and Technology Development Fund, Macau SAR (No. 0056/2020/AFJ, 0158/2019/A3).

Haoliang Yuan received the B.Sc. and M.Sc. degrees from the Hubei University, Wuhan, China, in 2009 and 2012, and the Ph.D. degree from the University of Macau, 2016. Currently, he is working at Macau University of Science and Technology.

References (48)

  • Bokai Cao et al.

    Tensor-based multi-view feature selection with applications to brain diseases

  • Mansheng Chen et al.

    Multi-view clustering in latent embedding space

    Proceedings of the AAAI conference on artificial intelligence

    (2020)
  • Xiaojun Chen, Joshua Zhexue Haung, Feiping Nie, Renjie Chen, and Qingyao Wu. A self-balanced min-cut algorithm for...
  • Yiuming Cheung et al.

    Local kernel regression score for selecting features of high-dimensional data

    IEEE Trans. Knowl. Data Eng.

    (2009)
  • Dengxin Dai et al.

    Ensemble projection for semi-supervised image classification

  • Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In Proceedings of the IEEE...
  • Yinfu Feng, Jun Xiao, Yueting Zhuang, and Xiaoming Liu. Adaptive unsupervised multi-view feature selection for visual...
  • Derek Greene et al.

    Practical solutions to the problem of diagonal dominance in kernel document clustering

  • Isabelle Guyon et al.

    An introduction to variable and feature selection

    J. Mach. Learn. Res.

    (2003)
  • Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. In Advances in Neural Information...
  • Chenping Hou et al.

    Multi-view unsupervised feature selection with adaptive similarity and view weight

    IEEE Trans. Knowl. Data Eng.

    (2017)
  • Shudong Huang et al.

    Measuring diversity in graph learning: A unified framework for structured multi-view clustering

    IEEE Trans. Knowl. Data Eng.

    (2021)
  • Hong Jia et al.

    Subspace clustering of categorical and numerical data with an unknown number of clusters

    IEEE Trans. Neural Networks Learn. Syst.

    (2017)
  • Meina Kan et al.

    Multi-view discriminant analysis

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2016)
  • Cited by (0)

    Haoliang Yuan received the B.Sc. and M.Sc. degrees from the Hubei University, Wuhan, China, in 2009 and 2012, and the Ph.D. degree from the University of Macau, 2016. Currently, he is working at Macau University of Science and Technology.

    Junyu Li received the B.Eng. and M.Eng. degrees from the Guangdong University of Technology, Guangzhou, China, in 2017 and 2020. Currently, he is pursuing the Ph.D. degree in South China University of Technology.

    Yong Liang received the B.S. and M.S. degrees in Applied Mathematics from Xi’an Jiaotong University, China, in 1996 and 1999, and the Ph.D. degree in Computer Science from the Chinese University of Hong Kong in 2003. He was with the Chinese University of Hong Kong (2004–2007) as a post doctor fellow. He is an Assistant Professor, Associate Professor and Professor at Macau University of Science and Technology. His research interests include machine learning, data mining, and bioinformatics.

    Yuan Yan Tang received the B.S. degree in electrical and computer engineering from Chongqing University, Chongqing, China, the M.Eng. degree in electrical engineering from the Beijing Institute of Post and Telecommunications, Beijing, China, and the Ph.D. degree in computer science from Concordia University, Montreal, QC, Canada. He is currently a Chair Professor with the Faculty of Science and Technology, University of Macau, Macau, China, and a Professor/Adjunct Professor/Honorary Professor with several institutes, including several universities in China, Concordia University, Canada, and Hong Kong Baptist University, Hong Kong. He has published over 400 technical papers and authored/co-authored over 25 monographs/books/bookchapters on subjects ranging from electrical engineering to computer science. His current research interests include wavelet theory and applications, pattern recognition, image processing, document processing, artificial intelligence, and Chinese computing. Dr. Tang is the Founder and the Editor-in-Chief of the International Journal on Wavelets, Multiresolution, and Information Processing, and an Associate Editor of several international journals, such as the International Journal on Pattern Recognition and Artificial Intelligence. He is the Founder and the Chair of Pattern Recognition Committee in the IEEE SYSTEMS, MAN, AND CYBERNETICS. He has served as the General Chair, the Program Chair, and a Committee Member for many international conferences, including the General Chair of the 18th International Conference on Pattern Recognition. He is the Founder and the General Chair of the series International Conferences on Wavelets Analysis and Pattern Recognition. He is a fellow of the International Associate of Pattern Recognition. 10

    1

    These authors have contributed equally to this work.

    View full text