Skip to main content
SpringerLink
Log in

Joint local structure preservation and redundancy minimization for unsupervised feature selection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Unsupervised feature selection is an indispensable pre-processing step in many data mining and pattern recognition tasks where the unlabeled high dimensional data are ubiquitous. Most of existing methods fail to explore the local geometric structure consistency (preservation) of the input data and minimize redundancy of selected features simultaneously. In this paper we propose a novel unsupervised feature selection method which jointly integrates the local geometric structure consistency and redundancy minimization (JLSPRM) into an unified framework. JLSPRM utilizes nonnegative spectral analysis to learn the cluster labels of the input data, then the local geometric structure consistency is developed to make the learned cluster labels more accurate, during which the feature selection operation is performed. To minimize the redundancy rate among selected features, the maximal information coefficient (MIC) is utilized to evaluate the correlation of the pairwise features. Besides, the ℓ2,1-norm is exerted on feature selection matrix which makes the framework decent for selecting features. An efficient iterative optimization algorithm is designed to obtain the solution of the unsupervised feature selection model. The superiority and effectiveness of our proposed approach over the state-of-the-art feature selection methods have also been validated through the extensive experiments on nine benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. In our experiments, we need normalize F to satisfy the orthogonal constraint \(\textbf {F}^{T}\textbf {F}=\textbf {I}_{c}\), i.e., \(F_{ij}=\frac {F_{ij}}{e_{j}}\), where ej is the j-th column norm of F, and the normalize operation will not give an affect on the convergence of F.

  2. http://yann.lecun.com/exdb/mnist/

  3. http://featureselection.asu.edu/datasets.php

  4. https://cs.nyu.edu/~roweis/data.html

  5. http://www.kasrl.org/jaffe.html

  6. http://archive.ics.uci.edu/ml/index.php

References

  1. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge, University Press

  2. Cai D, Chen X (2014) Large scale spectral clustering via landmark-based sparse representation. IEEE Trans Cybern 45(8):1669–1680

    Google Scholar 

  3. Cai D, Zhang C, He X (2010) Unsupervised feature selection for multi-cluster data. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 333–342

  4. Cai J, Luo J, Wang S, Yang S (2018) Feature selection in machine learning: a new perspective. Neurocomputing 300:70–79

    Google Scholar 

  5. Chen X, Yuan G, Wang W, Nie F, Chang X, Huang J (2018) Local adaptive projection framework for feature selection of labeled and unlabeled data. IEEE transactions on neural networks and learning systems (99):1–12

  6. Cheung YM, Zeng H (2009) Local kernel regression score for selecting features of high-dimensional data. IEEE Trans Knowl Data Eng 21(12):1798–1802

    Google Scholar 

  7. Constantinopoulos C, Titsias MK, Likas A (2006) Bayesian feature and model selection for gaussian mixture models. IEEE Trans Pattern Anal Mach Intell 28(6):1013–1018

    Google Scholar 

  8. Doquire G, Verleysen M (2013) A graph laplacian based approach to semi-supervised feature selection for regression problems. Neurocomputing 121:5–13

    Google Scholar 

  9. Dy JG, Brodley CE (2004) Feature selection for unsupervised learning. J Mach Learn Res 5:845–889

    MathSciNet  MATH  Google Scholar 

  10. Forman G (2003) An extensive empirical study of feature selection metrics for text classification. J Mach Learn Res 3:1289–1305

    MATH  Google Scholar 

  11. Gu B, Sheng VS (2016) A robust regularization path algorithm for v-support vector classification. IEEE Trans Neural Netw Learn Syst 28(5):1241–1248

    Google Scholar 

  12. Gui J, Sun Z, Ji S, Tao D, Tan T (2016) Feature selection based on structured sparsity: a comprehensive study. IEEE Trans Neural Netw Learn Syst 28(7):1490–1507

    MathSciNet  Google Scholar 

  13. He X, Cai D, Niyogi P (2006) Laplacian score for feature selection. In: Advances in neural information processing systems, pp 507–514

  14. Hou C, Nie F, Li X, Yi D, Wu Y (2014) Joint embedding learning and sparse regression: a framework for unsupervised feature selection. IEEE Trans Cybern 44(6):793–804

    Google Scholar 

  15. Hou C, Nie F, Tao H, Yi D (2017) Multi-view unsupervised feature selection with adaptive similarity and view weight. IEEE Trans Knowl Data Eng 29(9):1998–2011

    Google Scholar 

  16. Hou C, Nie F, Yi D, Tao D (2015) Discriminative embedded clustering: a framework for grouping high-dimensional data. IEEE Trans Neural Netw Learn Syst 26(6):1287–1299

    MathSciNet  Google Scholar 

  17. Huang J, Nie F, Huang H, Ding C (2014) Robust manifold nonnegative matrix factorization. ACM Trans Knowl Discov Data (TKDD) 8(3):11

    Google Scholar 

  18. Jain A, Zongker D (1997) Feature selection: evaluation, application, and small sample performance. IEEE Trans Pattern Anal Mach Intell 19(2):153–158

    Google Scholar 

  19. Kotsiantis S (2011) Feature selection for machine learning classification problems: a recent overview. Artif Intell Rev 42(1):157–176

    Google Scholar 

  20. Law MH, Figueiredo MA, Jain AK (2004) Simultaneous feature selection and clustering using mixture models. IEEE Trans Pattern Anal Mach Intell 26(9):1154–1166

    Google Scholar 

  21. Li X, Chen M, Wang Q (2019) Self-tuned discrimination-aware method for unsupervised feature selection. IEEE Trans Neural Netw Learn Syst 30(8):2275–2284

    MathSciNet  Google Scholar 

  22. Li Z, Liu J, Yang Y, Zhou X, Lu H (2014) Clustering-guided sparse structural learning for unsupervised feature selection. IEEE Trans Knowl Data Eng 26(9):2138–2150

    Google Scholar 

  23. Li Z, Yang Y, Liu J, Zhou X, Lu H (2012) Unsupervised feature selection using nonnegative spectral analysis. In: Twenty-sixth AAAI conference on artificial intelligence

  24. Liu J, Ji S, Ye J (2009) Multi-task feature learning via efficient ℓ2,1-norm minimization. In: Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence. AUAI Press, pp 339–348

  25. Ma Z, Nie F, Yang Y, Uijlings JR, Sebe N (2012) Web image annotation via subspace-sparsity collaborated feature selection. IEEE Trans Multimed 14(4):1021–1030

    Google Scholar 

  26. Maldonado S, Weber R (2009) A wrapper method for feature selection using support vector machines. Inf Sci 179(13):2208–2217

    Google Scholar 

  27. Mitra P, Murthy C, Pal SK (2002) Unsupervised feature selection using feature similarity. IEEE Trans Pattern Anal Mach Intell 24(3):301–312

    Google Scholar 

  28. Nie F, Huang H, Cai X, Ding C (2010) Efficient and robust feature selection via joint ℓ2,1-norms minimization. In: Advances in neural information processing systems, pp 1813–1821

  29. Nie F, Xiang S, Liu Y, Hou C, Zhang C (2012) Orthogonal vs. uncorrelated least squares discriminant analysis for feature extraction. Pattern Recogn Lett 33(5):485–491

    Google Scholar 

  30. Peng H, Liu CL (2019) Discriminative feature selection via employing smooth and robust hinge loss. IEEE Trans Neural Netw Learn Syst 30(3):788–802

    MathSciNet  Google Scholar 

  31. Peng H, Long F, Ding C (2005) Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence (8):1226–1238

  32. Qian M, Zhai C (2013) Robust unsupervised feature selection. In: Twenty-third international joint conference on artificial intelligence

  33. Reshef DN, Reshef YA, Finucane HK, Grossman SR, McVean G, Turnbaugh PJ, Lander ES, Mitzenmacher M, Sabeti PC (2011) Detecting novel associations in large data sets. Science 334 (6062):1518–1524

    MATH  Google Scholar 

  34. Shang R, Wang W, Stolkin R, Jiao L (2018) Non-negative spectral learning and sparse regression-based dual-graph regularized feature selection. IEEE Trans Cybern 48(2):793–806

    Google Scholar 

  35. Shang R, Zhang Z, Jiao L, Wang W, Yang S (2016) Global discriminative-based nonnegative spectral clustering. Pattern Recogn 55:172–182

    MATH  Google Scholar 

  36. Sheikhpour R, Sarram MA, Gharaghani S, Chahooki MAZ (2017) A survey on semi-supervised feature selection methods. Pattern Recogn 64:141–158

    MATH  Google Scholar 

  37. Shi J, Malik J (2000) Normalized cuts and image segmentation. Departmental Papers (CIS), pp 107

  38. Tang C, Liu X, Li M, Wang P, Chen J, Wang L, Li W (2018) Robust unsupervised feature selection via dual self-representation and manifold regularization. Knowl-Based Syst 145:109–120

    Google Scholar 

  39. Tao H, Hou C, Nie F, Jiao Y, Yi D (2015) Effective discriminative feature selection with nontrivial solution. IEEE Trans Neural Netw Learn Syst 27(4):796–808

    MathSciNet  Google Scholar 

  40. Teng L, Feng Z, Fang X, Teng S, Wang H, Kang P, Zhang Y (2019) Unsupervised feature selection with adaptive residual preserving. Neurocomputing 367:259–272

    Google Scholar 

  41. Wan Y, Chen X, Zhang J (2018) Global and intrinsic geometric structure embedding for unsupervised feature selection. Expert Syst Appl 93:134–142

    Google Scholar 

  42. Wang D, Nie F, Huang H (2015) Feature selection via global redundancy minimization. IEEE Trans Knowl Data Eng 27(10):2743–2755

    Google Scholar 

  43. Wang JJY, Yao J, Sun Y (2014) Semi-supervised local-learning-based feature selection. In: 2014 International joint conference on neural networks (IJCNN). IEEE, pp 1942–1948

  44. Wang Y, Feng L (2018) Hybrid feature selection using component co-occurrence based feature relevance measurement. Expert Syst Appl 102:83–99

    Google Scholar 

  45. Xu Z, King I, Lyu MRT, Jin R (2010) Discriminative semi-supervised feature selection via manifold regularization. IEEE Trans Neural Netw 21(7):1033–1047

    Google Scholar 

  46. Yan H, Yang J (2015) Sparse discriminative feature selection. Pattern Recogn 48(5):1827–1835

    Google Scholar 

  47. Yang J, Ong CJ (2012) An effective feature selection method via mutual information estimation. IEEE Trans Syst Man Cybern Part B (Cybern) 42(6):1550–1559

    Google Scholar 

  48. Yang Y, Shen HT, Ma Z, Huang Z, Zhou X (2011) ℓ2,1-norm regularized discriminative feature selection for unsupervised learning. In: Twenty-second international joint conference on artificial intelligence

  49. Yang Y, Shen HT, Nie F, Ji R, Zhou X (2011) Nonnegative spectral clustering with discriminative regularization. In: Twenty-fifth AAAI conference on artificial intelligence

  50. Yuan G, Chen X, Wang C, Nie F, Jing L (2018) Discriminative semi-supervised feature selection via rescaled least squares regression-supplement. In: Thirty-second AAAI conference on artificial intelligence

  51. Yuan H, Li J, Lai LL, Tang YY (2019) Joint sparse matrix regression and nonnegative spectral analysis for two-dimensional unsupervised feature selection. Pattern Recogn 89:119–133

    Google Scholar 

  52. Yuan H, Zheng J, Lai LL, Tang YY (2018) Sparse structural feature selection for multitarget regression. Knowl-Based Syst 160:200–209

    Google Scholar 

  53. Zeng H, Cheung YM (2011) Feature selection and kernel learning for local learning-based clustering. IEEE Trans Pattern Anal Mach Intell 33(8):1532–1547

    Google Scholar 

  54. Zhao Z, Liu H (2007) Spectral feature selection for supervised and unsupervised learning. In: Proceedings of the 24th international conference on Machine learning. ACM, pp 1151–1157

  55. Zhao Z, Wang L, Liu H (2010) Efficient spectral feature selection with minimum redundancy. In: Twenty-fourth AAAI conference on artificial intelligence

  56. Zhou N, Xu Y, Cheng H, Fang J, Pedrycz W (2016) Global and local structure preserving sparse subspace learning: an iterative approach to unsupervised feature selection. Pattern Recogn 53:87–101

    MATH  Google Scholar 

  57. Zhou T, Zhang C, Gong C, Bhaskar H, Yang J (2018) Multiview latent space learning with feature redundancy minimization. IEEE Transactions on Cybernetics

  58. Zhu P, Xu Q, Hu Q, Zhang C, Zhao H (2018) Multi-label feature selection with missing labels. Pattern Recogn 74:488–502

    Google Scholar 

  59. Zhu P, Zhu W, Hu Q, Zhang C, Zuo W (2017) Subspace clustering guided unsupervised feature selection. Pattern Recogn 66:364–374

    Google Scholar 

  60. Zhu X, Li X, Zhang S, Ju C, Wu X (2017) Robust joint graph sparse coding for unsupervised spectral feature selection. IEEE Trans Neural Netw Learn Syst 28(6):1263–1275

    MathSciNet  Google Scholar 

  61. Zhu X, Zhang S, Hu R, Zhu Y, et al. (2018) Local and global structure preservation for robust unsupervised spectral feature selection. IEEE Trans Knowl Data Eng 30(3):517–529

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. This article has been awarded by the National Natural Science Foundation of China (61941113), the Fundamental Research Fund for the Central Universities (30918015103, 30918012204), Nanjing Science and Technology Development Plan Project (201805036), and ”13th Five-Year” equipment field fund (61403120501), China Academy of Engineering Consulting Research Project (2019-ZD-1-02-02), National Social Science Foundation (18BTQ073), State Grid Technology Proj-ect (5211XT190033).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

    Hao Li, Yongli Wang, Peng Hu & Ruxin Zhao

  2. School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, China

    Yanchao Li

Authors
  1. Hao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongli Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanchao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruxin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongli Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Wang, Y., Li, Y. et al. Joint local structure preservation and redundancy minimization for unsupervised feature selection. Appl Intell 50, 4394–4411 (2020). https://doi.org/10.1007/s10489-020-01800-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01800-6

Keywords

Search

Navigation