Skip to main content
SpringerLink
Log in

Rotation-Based Ensemble Classifiers for High-Dimensional Data

  • Chapter
  • First Online:
Fusion in Computer Vision

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

Abstract

In past 20 years, Multiple Classifier System (MCS) has shown great potential to improve the accuracy and reliability of pattern classification. In this chapter, we discuss the major issues of MCS, including MCS topology, classifier generation, and classifier combination, providing a summary of MCS applied to remote sensing image classification, especially in high-dimensional data. Furthermore, the recently rotation-based ensemble classifiers, which encourage both individual accuracy and diversity within the ensemble simultaneously, are presented to classify high-dimensional data, taking hyperspectral and multidate remote sensing images as examples. Rotation-based ensemble classifiers project the original data into a new feature space using feature extraction and subset selection methods to generate the diverse individual classifiers. Two classifiers: Decision Tree (DT) and Support Vector Machine (SVM), are selected as the base classifier. Unsupervised and supervised feature extraction methods are employed in the rotation-based ensemble classifiers. Experimental results demonstrated that rotation-based ensemble classifiers are superior to Bagging, AdaBoost and random-based ensemble classifiers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Achlioptas, D (2001) Database-friendly random projections. In: Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on principles of database systems, New York, NY, USA, 2001, pp 274–281

    Google Scholar 

  2. Aksela M, Laaksonen J (2006) Using diversity of errors for selecting members of a committee classifier. Pattern Recogn 39(4):608–623

    Article  MATH  Google Scholar 

  3. Bakos KL, Gamba P (2011) Hierarchical hybrid decision tree fusion of multiple hyperspectral data processing chains. IEEE Trans Geosci Remote Sens 49(1–2):388–394

    Article  Google Scholar 

  4. Benediktsson JA, Chanussot J, Fauvel M (2007) Multiple classifier systems in remote sensing: from basics to recent developments. Lect Notes Comput Sci 4472:501–512

    Article  Google Scholar 

  5. Benediktsson JA, Sveinsson JR (1997) Hybrid consensus theoretic classification. IEEE Trans Geosci Remote Sens 35(4):833–843

    Article  Google Scholar 

  6. Benediktsson JA, Swain PH (1992) Consensus theoretic classification methods. IEEE Trans Syst Man Cybern 22(4):688–704

    Article  MATH  Google Scholar 

  7. Biggio B, Fumera G, Roli F (2010) Multiple classifier systems for robust classifier design in adversarial environments. J Mach Learn Cybern 1:27–41

    Article  Google Scholar 

  8. Braun AC, Weidner U, Hinz S (2012) Classification in high-dimensional feature spaces assessment using SVM, IVM and RVM with focus on simulated EnMap data. IEEE J Sel Top Appl Earth Observations Remote Sens 5(2):436–443

    Article  Google Scholar 

  9. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MATH  MathSciNet  Google Scholar 

  10. Breiman L (2001) Random forest. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  11. Briem G, Benediktsson J, Sveinsson J (2002) Multiple classifiers applied to multisource remote sensing data. IEEE Trans Geosci Remote Sens 40(10):2291–2299

    Article  Google Scholar 

  12. Brown G, Wyatt J, Harris R, Yao X (2005) Diversity creation methods: a survey and categorisation. Inf Fusion 6(1):5–20

    Article  Google Scholar 

  13. Bryll RK, Gutierrez-Osuna R, Quek FKH (2003) Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets. Pattern Recogn 36(6):1291–1302

    Article  MATH  Google Scholar 

  14. Chan JC, Paelinckx D (2008) Evaluation of Random Forest and AdBboost tree-based ensemble classification and spectral band selection for ecotope mapping using airborne hyperspectral imagery. Remote Sens Environ 112(6):2999–3011

    Article  Google Scholar 

  15. Chandra A, Yao X (2006) Evolving hybrid ensembles of learning machines for better generalisation. Neurocomputing 69(7–9):686–700

    Article  Google Scholar 

  16. Cheriyadat A, Bruce LM (2003) Why principal component analysis is not an appropriate feature extraction method for hyperspectral data. In: Proceedings of IEEE geoscience and remote sensing symposium (IGARSS), Toulouse, France, 2003, pp 3420–3422

    Google Scholar 

  17. Cunningham P, Carney J (2000) Diversity versus quality in classification ensembles based on feature selection. In: 11th European conference on machine learning, Barcelona, Spain, 2000, pp 109–116

    Google Scholar 

  18. Dell’Acqua F, Gamba P, Ferari A, Palmason BJA, Arnason K (2004) Exploiting spectral and spatial information in hyperspectral urban data with high resolution. IEEE Geosci Remote Sens Lett 1(4):322–326

    Article  Google Scholar 

  19. Dietterich TG, Bakiri G (1995) Solving multiclass learning problems via error-correcting output codes. J Artif Intell Res 2(1):263–286

    MATH  Google Scholar 

  20. Doan HTX, Foody GM (2007) Increasing soft classification accuracy through the use of an ensemble of classifiers. Int J Remote Sens 28(20):4609–4623

    Article  Google Scholar 

  21. Du P, Xia J, Zhang W, Tan K, Liu Y, Liu S (2012) Multiple classifier system for remote sensing image classification: a review. Sensors 12(4):4764–4792

    Article  Google Scholar 

  22. Du P, Zhang W, Xia J (2011) Hyperspectral remote sensing image classification based on decision level fusion. Chin Optics Lett 9(3):031002–031004

    Article  Google Scholar 

  23. Duca R, Frate FD (2008) Hyperspectral and multiangle CHRIS-PROBA images for the generation of land cover maps. IEEE Trans Geosci Remote Sens 10:2857–2866

    Article  Google Scholar 

  24. Fauvel M, Chanussot J, Benediktsson JA (2009) Kernel principal component analysis for the classification of hyperspectral remote sensing data over urban areas. EURASIP J Adv Signal Process 2:1–15

    Article  Google Scholar 

  25. Fleiss J (1981) Statistical methods for rates and proportions. Wiley, New York

    Google Scholar 

  26. Foody GM (2009) Classification accuracy comparison: hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority. Remote Sens Environ 113(8):1658–1663

    Article  Google Scholar 

  27. Foody GM, Boyd DS, Sanchez-Hernandez C (2007) Mapping a specific class with an ensemble of classifiers. Int J Remote Sens 28:1733–1746

    Article  Google Scholar 

  28. Freund Y, Schapire RE (1996) Experiments with a new Boosting algorithm. In: International conference on machine learning, Bari, Italy, 1996, pp 148–156

    Google Scholar 

  29. Gao J, Liang F, Fan W, Sun Y, Han J (2013) A graph-based consensus maximization approach for combining multiple supervised and unsupervised models. IEEE Trans Knowl Data Eng 25(1):15–28

    Article  Google Scholar 

  30. Giacinto G, Roli F (2001) Design of effective neural network ensembles for image classification. Image Vis Comput J 19(9/10):697–705

    Google Scholar 

  31. Giacinto G, Roli F, Fumera G (2000) Selection of image classifiers. Electron Lett 36:420–422

    Article  Google Scholar 

  32. Giacinto G, Roli F, Vernazza G (1997) Comparison and combination of statistical and neural network algorithms for remote-sensing image classification. Neurocomputation in remote sensing data analysis. Springer, Berlin, pp 117–124

    Google Scholar 

  33. Gislason PO, Benediktsson JA, Sveinsson JR (2006) Random forests for land cover classification. Pattern Recogn Lett 27(4):294–300

    Article  Google Scholar 

  34. Gmez-Chova L, Fernndez-Prieto D, Calpe-Maravilla J, Soria-Olivas E, Vila-Francs J, Camps-Valls G (2006) Urban monitoring using multi-temporal SAR and multi-spectral data. Pattern Recogn Lett 27(4):234–243

    Google Scholar 

  35. Guyon I, Gunn S, Nikravesh M, Zadeh L (2006) Feature extraction: foundations and applications. Springer, New York

    Google Scholar 

  36. Ham J, Chen Y, Crawford MM, Ghosh J (2005) Investigation of the random forest framework for classification of hyperspectral data. IEEE Trans Geosci Remote Sens 43(3):492–501

    Article  Google Scholar 

  37. Hansen LK, Salamon P (1990) Neural network ensembles. IEEE Trans Pattern Anal Mach Intell 12(10):993–1001

    Article  Google Scholar 

  38. He Z, Xu X, Deng S (2005) A cluster ensemble method for clustering categorical data. Inf Fusion 6(2):143–151

    Article  Google Scholar 

  39. Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):832–844

    Article  Google Scholar 

  40. Hughes G (1968) On the mean accuracy of statistical pattern recognizers. IEEE Trans Inf Theory 14(1):55–63

    Article  Google Scholar 

  41. Iam-On N, Boongoen T, Garrett S, Price C (2011) A link-based approach to the cluster ensemble problem. IEEE Trans Pattern Anal Mach Intell 33(12):2396–2409

    Article  Google Scholar 

  42. Richards JA, Jia X (2006) Remote sensing digital image analysis: an introduction. Springer, New York

    Google Scholar 

  43. Kang H-J, Lee S-W (2000) An information-theoretic strategy for constructing multiple classifier systems. In: International conference on pattern recognition (ICPR), Barcelona, Spain, 2000, pp 2483–2486

    Google Scholar 

  44. Kawaguchi K, Nishii R (2007) Hyperspectral image classification by bootstrap AdaBoost with random decision stumps. IEEE Trans Geosci Remote Sens 45(11–2):3845–3851

    Article  Google Scholar 

  45. Kohavi R, Wolpert DH (1996) Bias plus variance decomposition for zero-one loss functions. In: 13th international conference on machine learning (ICML), Bari, Italy, 1996, pp 275–283

    Google Scholar 

  46. Kong Z, Cai Z (2007) Advances of research in fuzzy integral for classifiers’ fusion. In: Proceedings of the eigth ACIS international conference on software engineering, artificial intelligence, networking, and parallel/distributed computing, Washington, DC, USA pp 809–814

    Google Scholar 

  47. Krogh A, Vedelsby J (1995) Neural network ensembles, cross validation, and active learning. Adv Neural Inf Process Syst 7:231–238

    Google Scholar 

  48. Kumar S, Ghosh J, Crawford MM (2002) Hierarchical fusion of multiple classifiers for hyperspectral data analysis. Pattern Anal Appl 5:210–220

    Article  MATH  MathSciNet  Google Scholar 

  49. Kuncheva L, Whitaker C, Shipp C, Duin R (2000) Is independence good for combining classifiers? In: International conference on pattern recognition (ICPR), 2000, p 2168

    Google Scholar 

  50. Kuncheva LI (2004) Combining pattern classifiers: methods and algorithms. Wiley-Interscience, New Jersey

    Google Scholar 

  51. Kuncheva LI (2005) Diversity in multiple classifier systems. Inf Fusion 6(1):3–4

    Article  MathSciNet  Google Scholar 

  52. Kuncheva LI, Rodriguez JJ (2007) An experimental study on rotation forest ensembles. In: Proceedings of the 7th international workshop on multiple classifier systems, Prague, Czech Republic, 2007, pp 459–468

    Google Scholar 

  53. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207

    Article  MATH  Google Scholar 

  54. Landgrebe DA (1984) Signal theory methods in multispectral remote sensing. Wiley, New York

    Google Scholar 

  55. Lawrence R, Bunna A, Powellb S, Zambon M (2004) Classification of remotely sensed imagery using stochastic gradient boosting as a refinement of classification tree analysis. Remote Sens Environ 90(3):331–336

    Article  Google Scholar 

  56. Lu D, Weng Q (2007) A survey of image classification methods and techniques for improving classification performance. Int J Remote Sens 28(5):823–870

    Article  Google Scholar 

  57. Margineantu DD, Dietterich TG (1997) Pruning adaptive boosting. In: Proceedings of the fourteenth international conference on machine learning, San Francisco, CA, USA, 1997, pp 211–218

    Google Scholar 

  58. Martinez-Munoz G, Suarez A (2005) Switching class labels to generate classification ensembles. Pattern Recogn 38(10):1483–1494

    Article  Google Scholar 

  59. Melgani F, Bruzzone L (2004) Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans Geosci Remote Sens 42(8):1778–1790

    Article  Google Scholar 

  60. Michail P, Benediktsson JA, Ioannis K (2002) The effect of classifier agreement on the accuracy of the combined classifier in decision level fusion. IEEE Trans Geosci Remote Sens 39(11):2539–2546

    Google Scholar 

  61. Mojaradi B, Abrishami-Moghaddam H, Zoej M, Duin R (2009) Dimensionality reduction of hyperspectral data via spectral feature extraction. IEEE Trans Geosci Remote Sens 47(7):2091–2105

    Article  Google Scholar 

  62. Moon H, Ahn H, Kodell RL, Baek S, Lin C-J, Chen JJ (2007) Ensemble methods for classification of patients for personalized medicine with high-dimensional data. Artif Intell Med 41(3):197–207

    Article  Google Scholar 

  63. Moreno-Seco F, Inesta J, Ponce De Leon P, Mico L (2006) Comparison of classifier fusion methods for classification in pattern recognition tasks. Lect Notes Comput Sci 4109:705–713

    Article  Google Scholar 

  64. Navalgund RR, Jayaraman V, Roy PS (2007) Remote sensing applications: an overview. Current 93(12):1747–1766

    Google Scholar 

  65. Nemmour H, Chibani Y (2006) Multiple support vector machines for land cover change detection: an application for mapping urban extensions. ISPRS J Photogrammetry Remote Sens 61(2):125–133

    Article  Google Scholar 

  66. Ozcift A, Gulten A (2011) Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms. Comput Methods Programs Biomed 104(3):443–451

    Article  Google Scholar 

  67. Pal M, Foody GM (2010) Feature selection for classification of hyperspectral data by SVM. IEEE Trans Geosci Remote Sens 48(5):2297–2307

    Article  Google Scholar 

  68. Partridge D, Krzanowski W (1997) Software diversity: practical statistics for its measurement and exploitation. Inf Softw Technol 39(10):707–717

    Article  Google Scholar 

  69. Plaza A, Martinez P, Plaza J, Perez R (2005) Dimensionality reduction and classification of hyperspectral image data using sequences of extended morphological transformations. IEEE Trans Geosci Remote Sens 43(3):466–479

    Article  Google Scholar 

  70. Rahman A, Fairhurst M (1999) Serial combination of multiple experts: a unified evaluation. Pattern Anal Appl 2:292–311

    Article  Google Scholar 

  71. Ranawana R, Palade V (2006) Multi-classifier systems: review and a roadmap for developers. Int J Hybrid Intell Syst 3(1):1–41

    Google Scholar 

  72. Rodriguez JJ, Kuncheva LI (2009) Rotation forest: a new classifier ensemble method. IEEE Trans Pattern Anal Mach Intell 28(10):1619–1630

    Article  Google Scholar 

  73. Rodriguez-Galianoa VF, Ghimireb B, Roganb J, Chica-Olmoa M, Rigol-Sanchezc JP (2011) An assessment of the effectiveness of a random forest classifier for land-cover classification. ISPRS J Photogrammetry Remote Sens 67:93–104

    Article  Google Scholar 

  74. Rokach L (2010) Pattern classification using ensemble methods. World Scientific, Singapore

    Google Scholar 

  75. Ruitenbeek F, Debba P, Meer F, Cudahy T, Meijde M, Hale M (2006) Mapping white micas and their absorption wavelengths using hyperspectral band ratios. Remote Sens Environ 102 (3–4):211–222

    Google Scholar 

  76. Ruta D, Gabrys B (2000) An overview of classifier fusion methods. Comput Inf Syst 7(1):1–10

    Google Scholar 

  77. Schölkopf B, Platt JC, Shawe-Taylor J, Smola AJ, Williamson RC (2001) Estimating the support of a high-dimensional distribution. Neural Comput 13(7):1443–1471

    Article  MATH  Google Scholar 

  78. Smits PC (2002) Multiple classifier systems for supervised remote sensing image classification based on dynamic classifier selection. IEEE Trans Geosci Remote Sens 40:801–813

    Article  Google Scholar 

  79. Steele BM (2000) Combining multiple classifiers: an application using spatial and remotely sensed information for land cover type mapping. Remote Sens Environ 74(3):545–556

    Article  Google Scholar 

  80. Sugiyama M (2007) Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. J Mach Learn Res 27(8):1021–1064

    Google Scholar 

  81. Sun Q, Ye XQ, Gu WK (2000) A new combination rules of evidence theory (in chinese). Acta Electronica Sinica 8(1):117–119

    Google Scholar 

  82. Tsoumakas G, Partalas I, Vlahavas I (2009) An ensemble pruning primer. In: Okun O, Valentini G (eds) Applications of supervised and unsupervised ensemble methods. Springer, Berlin, pp 1–13

    Google Scholar 

  83. Vapnik VN (1995) The nature of statistical learning theory. Springer, Berlin

    Google Scholar 

  84. Waske B, Benediktsson JA, Arnason K, Sveinsson JR (2009) Mapping of hyperspectral aviris data using machine-learning algorithms. Can J Remote Sens 35:106–116

    Article  Google Scholar 

  85. Waske B, Van Der Linden S, Benediktsson JA, Rabe A, Hostert P (2010) Sensitivity of support vector machines to random feature selection in classification of hyperspectral data. IEEE Trans Geosci Remote Sens 48(7):2880–2889

    Article  Google Scholar 

  86. Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82

    Article  Google Scholar 

  87. Woods K, Kegelmeyer WP, Bowyer K (1997) Combination of multiple classifiers using local accuracy estimates. IEEE Trans Pattern Anal Mach Intell 19(4):405–410

    Article  Google Scholar 

  88. Wozniak M, Grana M, Corchado E (2014) A survey of multiple classifier systems as hybrid systems. Inf Fusion 16(1):3–17

    Article  Google Scholar 

  89. Xia J, Du P, He X, Chanussot J (2014) Hyperspectral remote sensing image classification based on rotation forest. IEEE Geosci Remote Sens Lett 11:239–243

    Article  Google Scholar 

  90. Xu L, Krzyzak A, Suen CY (1992) Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans Syst Man Cybern 22(3):418–435

    Article  Google Scholar 

  91. Yang J-M, Kuo B-C, Yu P-T, Chuang C-H (2010) A dynamic subspace method for hyperspectral image classification. IEEE Trans Geosci Remote Sens 48(7):2840–2853

    Article  Google Scholar 

  92. Zeng H, Triosell HJ (2004) Dimensionality reduction in hyperspectral image classification. In: IEEE International Conference on Image Processing (ICIP), Singapore, 2004, pp 913–916

    Google Scholar 

  93. Zhang Y (2010) Ten years of technology advancement in remote sensing and the research in the crc-agip lab in gce. Geomatica 64(2):173–189

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. GIPSA-lab, Grenoble Institute of Technology, 38400, Grenoble, France

    Junshi Xia, Jocelyn Chanussot & Xiyan He

  2. Faculty of Electrical and Computer Engineering, University of Iceland, Reykjavik, Iceland

    Jocelyn Chanussot

  3. Department of Geographical Information Science, Nanjing University, Nanjing, 210093, China

    Peijun Du

Authors
  1. Junshi Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jocelyn Chanussot
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peijun Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiyan He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peijun Du .

Editor information

Editors and Affiliations

  1. University Politehnica of Bucharest, Romania

    Bogdan Ionescu

  2. University of Bordeaux, Talence, France

    Jenny Benois-Pineau

  3. Queen Mary University of London, London, United Kingdom

    Tomas Piatrik

  4. Lab. of Informatics of Grenoble, France

    Georges Quénot

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Xia, J., Chanussot, J., Du, P., He, X. (2014). Rotation-Based Ensemble Classifiers for High-Dimensional Data. In: Ionescu, B., Benois-Pineau, J., Piatrik, T., Quénot, G. (eds) Fusion in Computer Vision. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-319-05696-8_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-05696-8_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-05695-1

  • Online ISBN: 978-3-319-05696-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation