Skip to main content
SpringerLink
Log in

A fast SVM-based wrapper feature selection method driven by a fuzzy complementary criterion

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

The selection of informative and non-redundant features has become a prominent step in pattern classification. However, despite the intensive research, it is still an open issue to identify valuable feature subsets, especially in highly dimensional feature spaces. This paper proposes a wrapper feature selection method, in the context of support vector machines (SVMs), named Wr-SVM-FuzCoC. Our method combines effectively the advantages of the wrapper and filter approaches, achieving three goals simultaneously: classification performance, dimensionality reduction, and computational efficiency. In the filter part, a forward feature search methodology is developed, driven by a fuzzy complementary criterion, whereby at each iteration a feature is selected that exhibits the maximum additional contribution in regard to the previously selected subset. The quality of single features or feature subsets is assessed via a fuzzy local evaluation criterion with respect to patterns. This is achieved by the so-called fuzzy partition vector (FPV), comprising the fuzzy membership grades of every pattern in their target classes. Derivation of the feature FPVs is accomplished by incorporating a fuzzy output kernel-based support vector machine. The proposed method is favorably compared with existing SVM-based wrapper methods, in terms of performance capability and computational speed. Experimental investigation is carried out using a diverse pool of real datasets, including moderate and high-dimensional feature spaces.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Ben-Bassat M (1968) Pattern recognition and reduction of dimensionality. In: Krishnaiah PR, Kanal LN (eds) Handbook of statistics-II. North Holland, New York, pp 773–791

  2. Bonev B, Escolano F, Gazorla M (2008) Feature selection, mutual information, and classification of high-dimensional patterns. Pattern Anal Appl 11(3):309–319

    Article  MathSciNet  Google Scholar 

  3. Bradley P, Mangasarian O (1998) Feature selection via concave minimization and support vector machines. In: Machine learning proceedings of the 13th international conference on machine learning (ICML’98), San Francisco, CA, pp 82–90

  4. Burges CJC (1998) A tutorial on support vector machines for pattern recognition. Data Mining Knowl Discov 2(2):121–167

    Article  Google Scholar 

  5. Dash M, Liu H (1997) Feature selection for classification, intelligent data analysis. Ann Int J 1(3):131–156

    Google Scholar 

  6. Dash M, Choi K, Scheuermann P, Liu H (2002) Feature selection for clustering—a filter solution. In: Proceedings of the second international conference on data mining, pp 115–122

  7. Devijver PA, Kittler J (1982) Pattern recognition: a statistical approach. Prentice-Hall, Englewood

    MATH  Google Scholar 

  8. Dubois D, Prade H (1980) Fuzzy sets and systems: theory and applications. Academic Press, London

  9. Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157

    MATH  Google Scholar 

  10. Guyon I, Weston J, Barnhill S, Bapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46(1–3):389–422

    Article  MATH  Google Scholar 

  11. Hsu C-W, Lin C-J (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13(2):415–425

    Google Scholar 

  12. Inza I, Larranaga P, Blanco R, Cerrolaza AJ (2004) Filter versus wrapper gene selection approaches in DNA microarray domains. Artif Intell Med 31:91–103

    Google Scholar 

  13. Jain A, Zongker D (1997) Feature selection: evaluation, application, and small sample performance. IEEE Trans Pattern Anal Mach Intell 19(2):153–158

    Article  Google Scholar 

  14. Jain AK, Duin RPW, Mao J (2000) Statistical pattern recognition: a review. IEEE Trans Pattern Anal Mach Intell 22(1):4–37

    Article  Google Scholar 

  15. Kohavi R, John GH (1997) Wrappers for feature subset selection. Artif Intell 97(1-2):273–324

    Article  MATH  Google Scholar 

  16. Kuncheva LI (2004) Combining pattern classifiers: methods and algorithms. Wiley, New York

  17. Kwak N, Choi C-H (2002) Input feature selection for classification problems. IEEE Trans Neural Netw 13(1):143–159

    Google Scholar 

  18. Lai C, Reinders MJT, Wessels L (2006) Random subspace method for multivariate feature selection. Pattern Recognit Lett 27(10):1067–1076

    Google Scholar 

  19. Liu H, Motoda H (1998) Feature selection for knowledge discovery and data mining. Kluwer Academic, Boston

    Book  MATH  Google Scholar 

  20. Liu Yi, Zheng Yuan F (2006) FS_SFS: a novel feature selection method for support vector machines. Pattern Recognit 39(7):1333–1345

    Article  MATH  Google Scholar 

  21. Maldonado S, Weber R (2009) A wrapper method for feature selection using support vector machines. Inf Sci 179:2208–2217

    Article  Google Scholar 

  22. Mao KZ (2004) Feature subset selection for support vector machines through discriminative function pruning analysis. IEEE Trans Syst Man Cybern 34(1):60–67

    Article  Google Scholar 

  23. Melgani F, Bruzzone L (2004) Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans Geosci Remote Sens 42(8):1778–1790

    Article  Google Scholar 

  24. Miranda J, Montoya R, Weber R (2005) Linear penalization support vector machines for feature selection. Pattern Recognition and Machine Intelligence, LNCS, vol 3776. Springer, Berlin, pp 188–192

  25. Mitra P, Murthy CA, Pal SK (2002) Unsupervised feature selection using feature similarity. IEEE Trans Pattern Anal Mach Intell 24(3):301–312

    Article  Google Scholar 

  26. Moustakidis SP, Theocharis JB, Giakas G (2008) Subject recognition based on ground reaction force measurements of gait signals. IEEE Trans Syst Man Cybern B 38(6):1476–1485

    Article  Google Scholar 

  27. Müller KR, Mika S, Rätsch G, Tsuda K, Schölkopf B (2001) An introduction to kernel-based learning algorithms. IEEE Trans Neural Netw 12(2):181–201

    Article  Google Scholar 

  28. Pal M (2006) Support vector machine-based feature selection for land cover classification: a case study with DAIS hyperspectral data. Int J Remote Sens 27(14):2877–2894

    Article  Google Scholar 

  29. Pal M, Foody GM (2010) Feature selection for classification of hyperspectral data by SVM. IEEE Trans Geosci Remote Sens 48(5):2297–2307

    Article  Google Scholar 

  30. Peng H, Long F, Ding C (2005) Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238

    Article  Google Scholar 

  31. Platt J (2000) Probabilistic outputs for support vector machines and comparison to regularized likehood methods. In: Smola AJ, Bartlett P, Schoelkopf B, Schurmans D (eds) Advances in large margin classifiers, pp 61–74

  32. Punch WF, Goodman ED, Pei M, Chia-Sun L, Hovland P, Enbody R (1993) Further research on feature selection and classification using genetic algorithms. In: Proceedings of the international conference on genetic algorithms, pp 557–564

  33. Rakotomanojy A (2003) Variable selection using SVM-based criteria. J Mach Learn Res 3:1357–1370

    MathSciNet  Google Scholar 

  34. Raymer M, Punch W, Goodman E, Kuhn L, Jain AK (2000) Dimensionality reduction using genetic algorithms. IEEE Trans Evol Comput 4:164–171

    Article  Google Scholar 

  35. Shieh M-D, Yang C-C (2008) Multiclass SVM-RFE for product form feature selection. Expert Syst Appl 35:531–541

    Article  Google Scholar 

  36. Weston J, Mukherjee S, Chapelle O, Pontil M, Poggio T, Vapnik V (2001) Feature selection for SVMs, In Advances in neural information processing systems 13. MIT Press, Cambridge

    Google Scholar 

  37. Xiong MM, Fang X, Zhao JY (2001) Biomarker identification by feature wrappers. Genome Res 11:1878–1887

    Google Scholar 

  38. Xu L, Yan P, Chang T (1988) Best first strategy for feature selection. In: Proceedings of the ninth international conference on pattern recognition, pp 706–708

  39. UCI machine learning repository. (http://www.ics.uci.edu/~mlearn/MLRepository.html)

  40. Yang J, Honavar V (1998) Feature subset selection using a genetic algorithm. IEEE Intell Syst 13:44–49

    Article  Google Scholar 

  41. Zadrozny B, Elkan C (2002) Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pp 694–699

Download references

Author information

Authors and Affiliations

  1. Division of Electronics and Computer Engineering, Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, 54124, Thessaloniki, Greece

    S. P. Moustakidis & J. B. Theocharis

Authors
  1. S. P. Moustakidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. B. Theocharis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. B. Theocharis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Moustakidis, S.P., Theocharis, J.B. A fast SVM-based wrapper feature selection method driven by a fuzzy complementary criterion. Pattern Anal Applic 15, 379–397 (2012). https://doi.org/10.1007/s10044-012-0293-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-012-0293-7

Keywords

Search

Navigation