Skip to main content
SpringerLink
Log in

An ensemble-based method for linear feature extraction for two-class problems

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

In this paper we propose three variants of a linear feature extraction technique based on Adaboost for two-class classification problems. Unlike other feature extraction techniques, we do not make any assumptions about the distribution of the data. At each boosting step we select from a pool of linear projections the one that minimizes the weighted error. We propose three different variants of the feature extraction algorithm, depending on the way the pool of individual projections is constructed. Using nine real and two artificial data sets of different original dimensionality and sample size we compare the performance of the three proposed techniques with three classical techniques for linear feature extraction: Fisher linear discriminant analysis (FLD), Nonparametric discriminant analysis (NDA) and a recently proposed feature extraction method for heteroscedastic data based on the Chernoff criterion. Our results show that for data sets of relatively low-original dimensionality FLD appears to be both the most accurate and the most economical feature extraction method (giving just one-dimension in the case of two classes). The techniques based on Adaboost fare better than the classical techniques for data sets of large original dimensionality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. We assume that the reader is familiar with Adaboost although the feature extraction should be reproducible from the Fig. 3

  2. Functions from the PRTOOLS 3.1.7 toolbox [24] have been used for classifiers 1–3. For the SVM classifier we used the OSU SVM Classifier Matlab toolbox 3.00 that can be downloaded from http://www.ece.osu.edu/∼maj/osu_svm/.

  3. Full information about the standard deviations and the calculated confidence intervals can be found at http://www.cvc.uab.es/∼davidm/experiments.htm

References

  1. Hyvarinen A, Karhunen J, Oja E (2001) Independent component analysis. John Wiley and Sons

    Google Scholar 

  2. Friedman JH (1987) Explanatory projection pursuit. J Am Statistical Assoc 82:249–266

    Article  MATH  Google Scholar 

  3. Lee DD, Seung HS (1999) Learning the parts of objects with nonnegative matrix factorization. Nature 401:788–791

    Article  PubMed  Google Scholar 

  4. Fisher R (1936) The use of multiple measurements in taxonomic problems. Ann Eugenics 7:179–188

    Google Scholar 

  5. Fukunaga K, Mantock J (1983) Nonparametric discriminant analysis. IEEE T Pattern 5(6):671–678

    Article  MATH  Google Scholar 

  6. Loog M, Duin RPW (2004) Linear dimensionality reduction via a heteroscedastic extension of lda: The Chernoff criterion. IEEE T Pattern Anal 26(6):732–739

    Article  Google Scholar 

  7. McLachlan GJ (2004) Discriminant analysis and statistical pattern recognition. John Wiley and Sons, Inc, New York

    MATH  Google Scholar 

  8. RST, SLK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290:2323–2326

    Google Scholar 

  9. Tenenbaum JB, de Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2323

    Article  Google Scholar 

  10. Long PM, Vega VB (2003) Boosting and microarray data. Machine Learning 52:31–44

    Article  MATH  Google Scholar 

  11. Athitsos V, Alon J, Sclaroff S, Kollios G (2004) Boostmap: a method for efficient approximate similarity rankings. In: CVPR (2), 2004, pp 268–275

  12. Sirlantzis K, Hoque S, Fairhurst MC (2002) Trainable multiple classifier schemes for handwritten character recognition. In: Multiple classifier systems, 2002, pp 169–178

  13. Brown G, Yao X, Wyatt J, Wersing H, Sendhoff B (2002) Exploiting ensemble diversity for automatic feature extraction. In: Proc. of the 9th international conference on neural information processing (ICONIP’02), 2002, pp 1786–1790

  14. Kuncheva LI (2004) Combining pattern classifiers. John Wiley and Sons

    MATH  Google Scholar 

  15. Breiman L (1998) Arcing classifiers. Ann Stat 26(3):801–849

    Article  MATH  MathSciNet  Google Scholar 

  16. Kirby M, Sirovich L (1990) Application of the Karhunen-Loeve procedure for the characterization of human faces. IEEE T Pattern Anal 12(1):103–108

    Article  Google Scholar 

  17. Fukunaga K (1990) Introduction to statistical pattern recognition, 2nd edn. Academic Press, Boston

    MATH  Google Scholar 

  18. Bressan M, Vitria J (2003) Nonparametric discriminant analysis and nearest neighbor classification. Pattern Recogn Lett 24(15):2743–2749

    Article  Google Scholar 

  19. Schapire RE (1999) A brief introduction to boosting. In: IJCAI, 1999, pp 1401–1406

  20. Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. In: International conference on machine learning, 1996, pp 148–156

  21. Blake C, Merz C (1998) UCI repository of machine learning databases http://www.ics.uci.edu/∼mlearn/MLRepository.html

  22. Skurichina M (2001) Stabilizing weak classifiers, Ph.D. thesis, Delft University of Technology

  23. Martinez A, Benavente R (1998) The AR face database. Tech Rep 24, Computer Vision Center (June 1998)

  24. Duin RPW (2004) PRTOOLS v3.17, Tech. rep., Delft University of Technology. http://www.ph.tn.tudelft.nl/∼bob/PRTOOLS.html

Download references

Acknowledgements

This work is supported by MCYT grant TIC2003-00654, and FP2000-4960 Ministerio de Ciencia y Tecnologia, Spain.

Author information

Authors and Affiliations

  1. Computer Vision Center, Department Informàtica, Universitat Autònoma de Barcelona, Edifici O Bellaterra, 08193, Barcelona, Spain

    David Masip & Jordi Vitrià

  2. School of Informatics, University of Wales, Bangor Dean Street, Bangor, Gwynedd, LL57 1UT, United Kingdom

    Ludmila I. Kuncheva

Authors
  1. David Masip
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ludmila I. Kuncheva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jordi Vitrià
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Masip.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Masip, D., Kuncheva, L.I. & Vitrià, J. An ensemble-based method for linear feature extraction for two-class problems. Pattern Anal Applic 8, 227–237 (2005). https://doi.org/10.1007/s10044-005-0002-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-005-0002-x

Keywords

Search

Navigation