Multi-view unsupervised feature selection with tensor low-rank minimization
Introduction
In many applications, the objects are often represented by different modalities or features and each of them has specific physical meaning and statistic property. For example, the remote sensing technique can use various sensors to detect a target such as the image features and the electromagnetic wave features [32]. Different visual descriptors such as SIFT [31], HOG [10] and GIST [29] can be used to represent images. Although different views reflect the same object, the feature spaces could be different among multiple views, which is known as heterogeneous features. Traditional single-view methods can only handle multi-view data by simply concatenating multi-view features as a new single feature set. However, the drawback is that it may ignore the correlation information among multiple views, which may degrade the performance or fail to work. In order to acquire more comprehensive information of an object from multi-view data, it is necessary to discover the complementary information provided by different views and the inner consistency information among all views. To process this issue, many multi-views methods [40], [47], [27] have been proposed and made a contribution to many applications such as bioinformatics, hyperspectral remote sensing, and data recovery [5], [41], [20], [38], [35]. To integrate the multi-view data for clustering, Chen et al. [6] propose to learn a global structure and cluster the multi-view data in embedding space, while Kang et al. [21] propose to utilize the multi-view information by fusing partitions. By leveraging the consistency and diversity simultaneously, Huang et al. [18] propose a unified framework for structured multi-view clustering. Considering the effect of cluster size, Peng et al. [30] propose a multi-view clustering method without parameter selection on cluster size, which aims to learn a space with geometric consistency and cluster assignment consistency. To explore the nonlinear relationships and consider the imbalance among different views, Huang et al. [17] propose an auto-weighted multi-view clustering method that learns multi-view similarity relationships in kernel spaces. For solving the view-insufficiency issue, Huang et al. [16] propose to simultaneously recover the latent intact space from multiple insufficient views and discover the cluster structure from the intact space.
In the field of multi-view learning, multi-view data also brings the problem of a surge in the number of features including noise and redundancy. Multi-view feature selection methods, aiming to reduce the dimensionality of multiple views data and explore the correlation between views, have arisen considerable research interests in recent years [36], [11], [15]. Traditional graph-based multi-view unsupervised feature selection methods construct one common graph matrix to exploit the local structure of multi-view data. Currently, there exist two ways to construct the common graph structure as follows. One is to fuse multiple graphs into one common graph by linear weighted fusion, i.e., , where is the weight coefficient of v-th view and is the graph matrix of v-th view. However, in this way, the graph structure needs to be computed in advance and fixed during the embedding procedure. Hence, these pre-defined graphs are inappropriate to exploit the local structure of the multi-view features in the embedding space. The other way is to learn a common graph structure across all views, which assumes that all views share the identical graph structure . Nevertheless, the aim of these two ways is to yield one common graph matrix to represent the local structure of muti-view data, which emphasizes consistency too much while the diversities among different views are restricted, i.e., the strong consistency problem.
In this paper, we propose a tensor low-rank constrained graph embedding method for multi-view unsupervised feature selection, which considers both consistency and diversity compatible well. In Fig. 1, one can see that the strong consistency constraint makes all views share the same graph. If these graphs are stacked into a 3-order tensor , we find that the rank of its lateral slice is equal to 1. First of all, lateral slide is sizes and it embodies the correlations among different views and samples. Secondly, based on the observation, the rank of lateral slide is 1. It means that each view’s graph provides the same information without any other complementary information, i.e., diversity. To capture the consistency and complementary information of multi-view data, we impose a low-rank constraint on lateral slides. In other words, by introducing a low-rank constraint on lateral slice instead of limiting its rank to 1, we can preserve the consistent information between views and exploit the diversity information at the same time. Therefore, we consider to respectively compute the graph matrix for its corresponding view and adopt a tensor low-rank constraint on this graph tensor formed by different graph matrices. On the one hand, our method can obtain the different graph structures for different views to preserve the diversity information. On the other hand, the tensor low-rank constraint on graph tensor can explore the high-order consistency of multiple views. Hence, our work makes consistency and diversity compatible well. Our main contributions are summarized as follows:
(1) To exploit the complementary information, we consider computing respectively the graph matrix for each view in the embedding space. To capture the latent consistency across views, we impose a tensor low-rank constraint on a tensor, which is stacked by these learned graph matrices. Both diversity and consistency information can be well ensured in our proposed model.
(2) An effective algorithm is presented to solve the optimization problem, together with the theoretical analyses on its convergence and computational complexity. Experimental results on several multi-view databases demonstrate the effectiveness of our proposed method and surpass some state-of-the-art competitive methods.
The paper is organized as follows. In Section 2, we briefly review and discuss the related works. In Section 3, we propose a tensor low-rank constrained graph embedding method for multi-view unsupervised feature selection, together with the theoretical analysis of convergence and computational complexity. Experiments on several multi-view data sets are conducted in Section 4. Section 5 concludes the paper.
Section snippets
Related works
Unsupervised data dimensionality reduction techniques have arisen lots of research interests and can be divided into two categories, i.e., subspace learning [19], [44] and feature selection [34]. In this paper, we focus on the unsupervised feature selection methods, which are approximately divided into three groups: wrapper, filter, and embedded. Wrapper methods select the optimal feature set iteratively with respect to a predetermined feature evaluated function. The typical wrapper method is
Formulation
We propose a multi-view unsupervised feature selection model by using graph embedding technique. In our model, we construct the different graph structures for multiple views in the embedding space. Besides, a tensor low-rank constraint is imposed on the tensor data, which is stacked by these graph matrices, to capture the high-order consistency across views. Hence, our model is formulated as follow:
Data sets and experiment setup
Before conducting experiments, we first give the details of six databases in Table 1. We employ several competing unsupervised feature selection methods to evaluate the performance of our method, including PCAScore [2], SPEC [48], LapScor [14], NDFS [24], WMCFS [42], AUMFS [11], ASVW [15], CRV-DCL [33]. To examine the selected features are effective or not, we use the clustering result of all features as the baseline. In the clustering experiments, unsupervised feature selection methods give
Conclusion
This paper proposes a tensor low-rank constrained graph embedding model for multi-view unsupervised feature selection, named TLR-MUFS. By adaptively learning the graphs under tensor low-rank constraints, our proposed method can solve the problem of keeping the view-specific information and consistency information in the multi-view graphs simultaneously. The view-specific information is retained in the graph corresponding to each view, while the consistency information is captured by the tensor
CRediT authorship contribution statement
Haoliang Yuan: Conceptualization, Methodology, Formal analysis, Writing - original draft. Junyu Li: Data curation, Software, Validation, Writing - review & editing. Yong Liang: Supervision, Funding acquisition, Writing - review & editing. Yuan Yan Tang: Supervision, Funding acquisition, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
This research work was supported in part by a grant from the “Macao Young Scholars Program” (Project code: AM201915), in part by the National Nature Science Foundation of China under Grant 61903091 and 62172458, in part by Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010801), and in part by the Science and Technology Development Fund, Macau SAR (No. 0056/2020/AFJ, 0158/2019/A3).
Haoliang Yuan received the B.Sc. and M.Sc. degrees from the Hubei University, Wuhan, China, in 2009 and 2012, and the Ph.D. degree from the University of Macau, 2016. Currently, he is working at Macau University of Science and Technology.
References (48)
- et al.
Multi-view intact space clustering
Pattern Recogn.
(2019) - et al.
Auto-weighted multi-view clustering via kernelized graph learning
Pattern Recogn.
(2019) - et al.
Beyond low-rank representations: Orthogonal clustering basis reconstruction with optimized graph structure for multi-view spectral clustering
Neural Networks
(2018) - et al.
Adaptive latent similarity learning for multi-view clustering
Neural Networks
(2020) - et al.
A new feature selection method for gaussian mixture clustering
Pattern Recogn.
(2009) - et al.
Multi-view learning overview: Recent progress and new challenges
Inform. Fusion
(2017) - Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances...
- Christopher Bishop. Neural Networks for Pattern Recognition. Oxford University Press, January...
- et al.(2004)
- et al.
A singular value thresholding algorithm for matrix completion
SIAM J. Optim.
(2010)
Tensor-based multi-view feature selection with applications to brain diseases
Multi-view clustering in latent embedding space
Proceedings of the AAAI conference on artificial intelligence
Local kernel regression score for selecting features of high-dimensional data
IEEE Trans. Knowl. Data Eng.
Ensemble projection for semi-supervised image classification
Practical solutions to the problem of diagonal dominance in kernel document clustering
An introduction to variable and feature selection
J. Mach. Learn. Res.
Multi-view unsupervised feature selection with adaptive similarity and view weight
IEEE Trans. Knowl. Data Eng.
Measuring diversity in graph learning: A unified framework for structured multi-view clustering
IEEE Trans. Knowl. Data Eng.
Subspace clustering of categorical and numerical data with an unknown number of clusters
IEEE Trans. Neural Networks Learn. Syst.
Multi-view discriminant analysis
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (0)
Haoliang Yuan received the B.Sc. and M.Sc. degrees from the Hubei University, Wuhan, China, in 2009 and 2012, and the Ph.D. degree from the University of Macau, 2016. Currently, he is working at Macau University of Science and Technology.
Junyu Li received the B.Eng. and M.Eng. degrees from the Guangdong University of Technology, Guangzhou, China, in 2017 and 2020. Currently, he is pursuing the Ph.D. degree in South China University of Technology.
Yong Liang received the B.S. and M.S. degrees in Applied Mathematics from Xi’an Jiaotong University, China, in 1996 and 1999, and the Ph.D. degree in Computer Science from the Chinese University of Hong Kong in 2003. He was with the Chinese University of Hong Kong (2004–2007) as a post doctor fellow. He is an Assistant Professor, Associate Professor and Professor at Macau University of Science and Technology. His research interests include machine learning, data mining, and bioinformatics.
Yuan Yan Tang received the B.S. degree in electrical and computer engineering from Chongqing University, Chongqing, China, the M.Eng. degree in electrical engineering from the Beijing Institute of Post and Telecommunications, Beijing, China, and the Ph.D. degree in computer science from Concordia University, Montreal, QC, Canada. He is currently a Chair Professor with the Faculty of Science and Technology, University of Macau, Macau, China, and a Professor/Adjunct Professor/Honorary Professor with several institutes, including several universities in China, Concordia University, Canada, and Hong Kong Baptist University, Hong Kong. He has published over 400 technical papers and authored/co-authored over 25 monographs/books/bookchapters on subjects ranging from electrical engineering to computer science. His current research interests include wavelet theory and applications, pattern recognition, image processing, document processing, artificial intelligence, and Chinese computing. Dr. Tang is the Founder and the Editor-in-Chief of the International Journal on Wavelets, Multiresolution, and Information Processing, and an Associate Editor of several international journals, such as the International Journal on Pattern Recognition and Artificial Intelligence. He is the Founder and the Chair of Pattern Recognition Committee in the IEEE SYSTEMS, MAN, AND CYBERNETICS. He has served as the General Chair, the Program Chair, and a Committee Member for many international conferences, including the General Chair of the 18th International Conference on Pattern Recognition. He is the Founder and the General Chair of the series International Conferences on Wavelets Analysis and Pattern Recognition. He is a fellow of the International Associate of Pattern Recognition. 10
- 1
These authors have contributed equally to this work.