Skip to main content
SpringerLink
Log in

Supervised feature selection algorithm via discriminative ridge regression

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

This paper studies a new feature selection method for data classification that efficiently combines the discriminative capability of features with the ridge regression model. It first sets up the global structure of training data with the linear discriminant analysis that assists in identifying the discriminative features. And then, the ridge regression model is employed to assess the feature representation and the discrimination information, so as to obtain the representative coefficient matrix. The importance of features can be calculated with this representative coefficient matrix. Finally, the new subset of selected features is applied to a linear Support Vector Machine for data classification. To validate the efficiency, sets of experiments are conducted with twenty benchmark datasets. The experimental results show that the proposed approach performs much better than the state-of-the-art feature selection algorithms in terms of the evaluating indicator of classification. And the proposed feature selection algorithm possesses a competitive performance compared with existing feature selection algorithms with regard to the computational cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

Notes

  1. http://archive.ics.uci.edu/ml/

  2. http://featureselection.asu.edu/datasets.php

References

  1. Alalga, A., Benabdeslem, K., Taleb, N.: Soft-constrained laplacian score for semi-supervised multi-label feature selection. Knowl. Inf. Syst. 47(1), 75–98 (2016)

    Article  Google Scholar 

  2. Bellal, F., Elghazel, H., Aussem, A.: A semi-supervised feature ranking method with ensemble learning. Pattern Recogn. Lett. 33(10), 1426–1433 (2012)

    Article  Google Scholar 

  3. Borchani, H., Varando, G., Bielza, C., Larrañaga, P.: A survey on multi-output regression. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 5(5), 216–233 (2015)

    Article  Google Scholar 

  4. Chen, L., Huang, J.Z.: Sparse reduced-rank regression for simultaneous dimension reduction and variable selection. J. Amer. Stat. Assoc. 107(500), 1533–1545 (2012)

    Article  MathSciNet  Google Scholar 

  5. Cheng, D., Zhang, S., Liu, X., Sun, K., Zong, M.: Feature selection by combining subspace learning with sparse representation. Multimed. Syst. 23(3), 1–7 (2017)

    Article  Google Scholar 

  6. Deng, Z., Zhu, X., Cheng, D., Zong, M., Zhang, S.: Efficient k nn classification algorithm for big data. Neurocomputing 195(C), 143–148 (2016)

    Article  Google Scholar 

  7. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern classification. Wiley, New Jersey (2012)

    MATH  Google Scholar 

  8. Gao, L., Song, J., Liu, X., Shao, J., Liu, J., Shao, J.: Learning in high-dimensional multimedia data: the state of the art. Multimed. Syst. 23(3), 303–313 (2017)

    Article  Google Scholar 

  9. Gao, L., Wang, Y., Li, D., Shao, J., Song, J.: Real-time social media retrieval with spatial, temporal and social constraints. Neurocomputing 253, 77–88 (2017)

    Article  Google Scholar 

  10. Germain, F.G., Mysore, G.J.: Stopping criteria for non-negative matrix factorization based supervised and semi-supervised source separation. IEEE Signal Process. Lett. 21(10), 1284–1288 (2014)

    Article  Google Scholar 

  11. Gu, Q., Li, Z., Han, J.: Generalized fisher score for feature selection. arXiv:1202.3725 (2012)

  12. Guo, Q., Wu, W., Massart, D., Boucon, C., De Jong, S.: Feature selection in principal component analysis of analytical data. Chemometr. Intell. Lab. Syst. 61 (1), 123–132 (2002)

    Article  Google Scholar 

  13. He, X., Cai, D., Niyogi, P.: Laplacian score for feature selection. In: Advances in Neural Information Processing Systems, pp. 507–514 (2005)

  14. Hu, R., Zhu, X., Cheng, D., He, W., Yan, Y., Song, J., Zhang, S.: Graph self-representation method for unsupervised feature selection. Neurocomputing 220, 130–137 (2017)

    Article  Google Scholar 

  15. Huang, H., Feng, H., Peng, C.: Complete local fisher discriminant analysis with laplacian score ranking for face recognition. Neurocomputing 89, 64–77 (2012)

    Article  Google Scholar 

  16. Izenman, A.J.: Linear discriminant analysis. In: Modern Multivariate Statistical Techniques, pp. 237–280. Springer, Berlin (2013)

    Google Scholar 

  17. Li, Z., Yang, Y., Liu, J., Zhou, X., Lu, H., et al.: Unsupervised feature selection using nonnegative spectral analysis. AAAI 2, 1026–1032 (2012)

    Google Scholar 

  18. Liu, B., Fang, B., Liu, X., Chen, J., Huang, Z., He, X.: Large margin subspace learning for feature selection. Pattern Recogn. 46(10), 2798–2806 (2013)

    Article  Google Scholar 

  19. Ng, A.Y.: Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In: Proceedings of the Twenty-first International Conference on Machine Learning, p 78. ACM (2004)

  20. Nie, F., Huang, H., Cai, X., Ding, C.H.: Efficient and robust feature selection via joint l 2,1-norms minimization. In: Advances in Neural Information Processing Systems, pp. 1813–1821 (2010)

  21. Peng, X., Yu, Z., Yi, Z., Tang, H.: Constructing the l2-graph for robust subspace learning and subspace clustering. IEEE Trans. Cybern. 47(4), 1053–1066 (2017)

    Article  Google Scholar 

  22. Pierre, C.: Semi-supervised feature selection via spectral analysis (2007)

  23. Qin, Y., Zhang, S., Zhu, X., Zhang, J., Zhang, C.: Semi-parametric optimization for missing data imputation. Appl. Intell. 27(1), 79–88 (2007)

    Article  Google Scholar 

  24. Song, J., Gao, L., Nie, F., Shen, H.T., Yan, Y., Sebe, N.: Optimized graph learning using partial tags and multiple features for image and video annotation. IEEE Trans. Image Process. 25(11), 4999–5011 (2016)

    Article  MathSciNet  Google Scholar 

  25. Song, J., Gao, L., Zou, F., Yan, Y., Sebe, N.: Deep and fast: Deep learning hashing with semi-supervised graph construction. Image Vis. Comput. 55, 101–108 (2016)

    Article  Google Scholar 

  26. Song, J., Shen, H.T., Wang, J., Huang, Z., Sebe, N., Wang, J.: A distance-computation-free search scheme for binary code databases. IEEE Trans. Multimed. 18(3), 484–495 (2016)

    Article  Google Scholar 

  27. Song, J., Yang, Y., Huang, Z., Shen, H.T., Luo, J.: Effective multiple feature hashing for large-scale near-duplicate video retrieval. IEEE Trans. Multimed. 15(8), 1997–2008 (2013)

    Article  Google Scholar 

  28. Trivedi, S., Pardos, Z.A., Heffernan, N.T.: Clustering students to generate an ensemble to improve standard test score predictions. In: International Conference on Artificial Intelligence in Education, pp. 377–384. Springer (2011)

  29. Wang, L., Zhu, J., Zou, H.: Hybrid huberized support vector machines for microarray classification. In: Proceedings of the 24th International Conference on Machine Learning, pp. 983–990. ACM, New York (2007)

  30. Wang, S., Nie, F., Chang, X., Yao, L., Li, X., Sheng, Q.Z.: Unsupervised feature analysis with class margin optimization. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 383–398. Springer, Berlin (2015)

    Chapter  Google Scholar 

  31. Ye, J.: Least squares linear discriminant analysis. In: Proceedings of the 24th international conference on Machine learning, pp. 1087–1093. ACM, New York (2007)

  32. Zeng, Z., Wang, X., Zhang, J., Wu, Q.: Semi-supervised feature selection based on local discriminative information. Neurocomputing 173(P1), 102–109 (2016)

    Article  Google Scholar 

  33. Zhang, S.: Shell-neighbor method and its application in missing data imputation. Appl. Intell. 35(1), 123–133 (2011)

    Article  Google Scholar 

  34. Zhang, S., Jin, Z., Zhu, X.: Missing data imputation by utilizing information within incomplete instances. J. Syst. Softw. 84(3), 452–459 (2011)

    Article  Google Scholar 

  35. Zhang, S., Li, X., Zong, M., Zhu, X., Cheng, D.: Learning k for knn classification. ACM Trans. Intell. Syst. Technol. 8(3), 43 (2017)

    Google Scholar 

  36. Zhang, S., Li, X., Zong, M., Zhu, X., Wang, R.: Efficient knn classification with different numbers of nearest neighbors. IEEE Transactions on Neural Networks and Learning Systems (2017). http://dx.doi.org/https://doi.org/10.1109/TNNLS.2017.2673241

    Article  MathSciNet  Google Scholar 

  37. Zhu, P., Zuo, W., Zhang, L., Hu, Q., Shiu, S.C.: Unsupervised feature selection by regularized self-representation. Pattern Recogn. 48(2), 438–446 (2015)

    Article  Google Scholar 

  38. Zhu, X., Huang, Z., Shen, H.T., Zhao, X.: Linear cross-modal hashing for efficient multimedia search. In: ACM MM, pp. 143–152 (2013)

  39. Zhu, X., Huang, Z., Yang, Y., Shen, H.T., Xu, C., Luo, J.: Self-taught dimensionality reduction on the high-dimensional small-sized data. Pattern Recogn. 46 (1), 215–229 (2013)

    Article  Google Scholar 

  40. Zhu, X., Li, X., Zhang, S.: Block-row sparse multiview multilabel learning for image classification. IEEE Trans. Cybern. 46(2), 450–461 (2016)

    Article  Google Scholar 

  41. Zhu, X., Li, X., Zhang, S., Ju, C., Wu, X.: Robust joint graph sparse coding for unsupervised spectral feature selection. IEEE Trans. Neural Netw. Learn. Syst. 28 (6), 1263–1275 (2017)

    Article  MathSciNet  Google Scholar 

  42. Zhu, X., Li, X., Zhang, S., Xu, Z., Yu, L., Wang, C.: Graph pca hashing for similarity search. IEEE Transactions on Multimedia (2017). https://doi.org/10.1109/TMM.2017.2703636

    Article  Google Scholar 

  43. Zhu, X., Suk, H., Wang, L., Lee, S., Shen, D.: A novel relational regularization feature selection method for joint regression and classification in AD diagnosis. Med. Image Anal. 38, 205–214 (2017)

    Article  Google Scholar 

  44. Zhu, X., Suk, H.-I., Huang, H., Shen, D.: Low-rank graph-regularized structured sparse regression for identifying genetic biomarkers. IEEE Transactions on Big Data (2017). https://doi.org/10.1109/TBDATA.2017.2735991

    Article  Google Scholar 

  45. Zhu, X., Suk, H.-I., Lee, S.-W., Shen, D.: Subspace regularized sparse multitask learning for multiclass neurodegenerative disease identification. IEEE Trans. Biomed. Eng. 63(3), 607–618 (2016)

    Article  Google Scholar 

  46. Zhu, X., Suk, H.-I., Shen, D.: A novel matrix-similarity based loss function for joint regression and classification in ad diagnosis. NeuroImage 100, 91–105 (2014)

    Article  Google Scholar 

  47. Zhu, X., Zhang, L., Huang, Z.: A sparse embedding and least variance encoding approach to hashing. IEEE Trans. Image Process. 23(9), 3737–3750 (2014)

    Article  MathSciNet  Google Scholar 

  48. Zhu, Y., Lucey, S.: Convolutional sparse coding for trajectory reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 529–540 (2015)

    Article  Google Scholar 

  49. Zhu, Y., Zhu, X., Kim, M., Shen, D., Wu, G.: Early diagnosis of alzheimers disease by joint feature selection and classification on temporally structured support vector machine. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 264–272 (2016)

    Chapter  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the China Key Research Program (Grant No: 2016YFB1000905), the Nation Natural Science Foundation of China (Grants No: 61573270, 61672177, and 61363009), National Association of public funds, the Guangxi Natural Science Foundation (Grant No: 2015GXNSFCB139011), the Guangxi High Institutions Program of Introducing 100 High-Level Overseas Talents, the Guangxi Collaborative Innovation Center of Multi-Source Information Integration and Intelligent Processing, the Research Fund of Guangxi Key Lab of MIMS (16-A-01-01 and 16-A-01-02), and the Guangxi Bagui Teams for Innovation and Research.

Author information

Authors and Affiliations

  1. Guangxi Key Lab of Multi-source Information Mining & Security, Guangxi Normal University, Guilin, Guangxi, 541004, China

    Shichao Zhang, Debo Cheng, Rongyao Hu & Zhenyun Deng

  2. Information Technology and Mathematical Sciences, University of South Australia, Adelaide, Australia

    Debo Cheng

Authors
  1. Shichao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Debo Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rongyao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhenyun Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shichao Zhang.

Additional information

This article belongs to the Topical Collection: Special Issue on Deep Mining Big Social Data

Guest Editors: Xiaofeng Zhu, Gerard Sanroma, Jilian Zhang, and Brent C. Munsell

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Cheng, D., Hu, R. et al. Supervised feature selection algorithm via discriminative ridge regression. World Wide Web 21, 1545–1562 (2018). https://doi.org/10.1007/s11280-017-0502-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-017-0502-9

Keywords

Search

Navigation