Skip to main content
SpringerLink
Log in

Feature Selection by Nonparametric Bayes Error Minimization

  • Conference paper
Advances in Knowledge Discovery and Data Mining (PAKDD 2008)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 5012))

Included in the following conference series:

Abstract

This paper presents an algorithmic framework for feature selection, which selects a subset of features by minimizing the nonparametric Bayes error. A set of existing algorithms as well as new ones can be derived naturally from this framework. For example, we show that the Relief algorithm greedily attempts to minimize the Bayes error estimated by k-Nearest-Neighbor method. This new interpretation not only reveals the secret behind Relief but also offers various opportunities to improve it or to establish new alternatives. In particular, we develop a new feature weighting algorithm, named Parzen-Relief, which minimizes the Bayes error estimated by Parzen method. Additionally, to enhance its ability to handle imbalanced and multiclass data, we integrate the class distribution with the max-margin objective function, leading to a new algorithm, named MAP-Relief. Comparison on benchmark data sets confirms the effectiveness of the proposed algorithms.

This work is supported in part by NSFC (#60275025, #60121302).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)

    MATH  Google Scholar 

  2. Carneiro, G., Vasconcelos, N.: Minimum Bayes Error Features for Visual Recognition by Sequential Feature Selection and Extraction. In: Proc. of CRV (2005)

    Google Scholar 

  3. Choi, E.: Feature Extraction Based on the Bhattacharyya Distance. Pattern Recognition 36, 1703–1709 (2003)

    Article  Google Scholar 

  4. Dash, M., Liu, H.: Feature Selection for Classification. Intelligent Data Analysis 1, 1131–1156 (1997)

    Article  Google Scholar 

  5. Gilad-Bachrach, R., Navot, A., Tishby, N.: Margin Based Feature Selection - Theory and Algorithms. In: Proc. of 21th ICML (2004)

    Google Scholar 

  6. Guyon, I., Elissee, A.: An Introduction to Variable and Feature Selection. JMLR 3, 1157–1182 (2003)

    Article  MATH  Google Scholar 

  7. Hild II, K.E., Erdogmus, D., Torkkola, K., Principe, C.: Feature Extraction Using Information-Theoretic Learning. IEEE Trans. PAMI 28(9), 1385–1392 (2006)

    Google Scholar 

  8. Jain, A., Zongker, D.: Feature Selection: Evaluation, Application, and Small Sample Performance. IEEE Trans. PAMI 19(2), 153–158 (1997)

    Google Scholar 

  9. Kira, K., Rendell, L.A.: A Practical Approach to Feature Selection. In: Proc. of 9th ICML, pp. 249–256 (1992)

    Google Scholar 

  10. Koller, D., Sahami, M.: Toward Optimal Feature Selection. In: Proc. of 13th ICML (1996)

    Google Scholar 

  11. Kononenko, I.: Estimating Attributes: Analysis and Extensions of RELIEF. In: Bergadano, F., De Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 171–182. Springer, Heidelberg (1994)

    Google Scholar 

  12. Mangasarian, O.L., Musicant, D.R.: Lagrangian Support Vector Machines. JMLR 1, 161–177 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  13. Robnik-Šikonja, M., Kononenko, I.: Theoretical and Empirical Analysis of ReliefF and RRlief. Machine Learning 53(1-2), 23–69 (2003)

    Article  Google Scholar 

  14. Saon, G., Padmanabhan, M.: Minimum Bayes Error Feature Selection for Continuous Speech Recognition. In: Proc. of NIPS (2002)

    Google Scholar 

  15. Sun, Y.J.: Iterative Relief for Feature Weighting: Algorithms, Theories, and Applications. IEEE Trans. PAMI 29(6), 1035–1051 (2007)

    Google Scholar 

  16. Vapnik, V.: Statistical Learning Theory. John Wiley & Sons, Chichester (1998)

    MATH  Google Scholar 

  17. Vasconcelos, N.: Feature Selection by Maximum Marginal Diversity: Optimality and Implications for Visual Recognition. In: Proc. of IEEE CVPR (2003)

    Google Scholar 

  18. Weston, J., Elisseeff, A., Schölkopf, B., Tipping, M.: Use of Zero-Norm with Linear Models and Kernel Mothods. JMLR 3, 1439–1461 (2003)

    Article  MATH  Google Scholar 

  19. Xuan, G., Zhu, X., Chai, P., Shi, Y., Fu, D.: Feature Selection Based on the Bhattacharyya Distance. In: Proc. of ICPR (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Lab of Pattern Recognition(NLPR) & Sino-French IT Lab(LIAMA), Institute of Automation, Chinese Academy of Sciences,  

    Shuang-Hong Yang & Bao-Gang Hu

  2. Graduate School, Chinese Academy of Sciences, P.O. Box 2728, Beijing, 100080, China

    Shuang-Hong Yang & Bao-Gang Hu

Authors
  1. Shuang-Hong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bao-Gang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Takashi Washio Einoshin Suzuki Kai Ming Ting Akihiro Inokuchi

Rights and permissions

Reprints and permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Yang, SH., Hu, BG. (2008). Feature Selection by Nonparametric Bayes Error Minimization. In: Washio, T., Suzuki, E., Ting, K.M., Inokuchi, A. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2008. Lecture Notes in Computer Science(), vol 5012. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68125-0_37

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-68125-0_37

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-68124-3

  • Online ISBN: 978-3-540-68125-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation