Skip to main content
SpringerLink
Log in
Book cover

Machine Learning in Document Analysis and Recognition pp 361–386Cite as

Review of Classifier Combination Methods

  • Chapter

Part of the book series: Studies in Computational Intelligence ((SCI,volume 90))

Classifier combination methods have proved to be an effective tool to increase the performance of pattern recognition applications. In this chapter we review and categorize major advancements in this field. Despite a significant number of publications describing successful classifier combination implementations, the theoretical basis is still missing and achieved improvements are inconsistent. By introducing different categories of classifier combinations in this review we attempt to put forward more specific directions for future theoretical research.We also introduce a retraining effect and effects of locality based training as important properties of classifier combinations. Such effects have significant influence on the performance of combinations, and their study is necessary for complete theoretical understanding of combination algorithms.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. . Cooke, R.: Experts in Uncertainty: Opinion and Subjective Probability in Sci- ence. Oxford University Press (1991)

    Google Scholar 

  2. Clemen, R., Winkler, R.: Combining probability distributions from experts in risk analysis. Risk Analysis 19 (1999) 187-203

    Google Scholar 

  3. Xu, L., Krzyzak, A., Suen, C.Y.: Methods for combining multiple classifiers and their applications to handwriting recognition. IEEE transactions on System, Man, and Cybernetics 23(3) (1992) 418-435

    Article  Google Scholar 

  4. Gader, P., Mohamed, M., Keller, J.: Fusion of Handwritten Word Classifiers. Pattern Recognition Letters 17 (1996) 577-584

    Article  Google Scholar 

  5. Sirlantzis, K., Hoque, S., Fairhurst, M.C.: Trainable Multiple Classifier Schemes for Handwritten Character Recognition. In: 3rd International Workshop on Multiple Classifier Systems (MCS), Cagliari, Italy, Lecture Notes in Computer Science, Springer-Verlag (2002) 169-178

    Chapter  Google Scholar 

  6. Wang, W., Brakensiek, A., Rigoll, G.: Combination of Multiple Classifiers for Handwritten Word Recognition. In: Proc. of the 8th International Workshop on Frontiers in Handwriting Recognition (IWFHR-8), Niagara-on-the-Lake, Canada (2002) 117-122

    Chapter  Google Scholar 

  7. .Lee, D.S.: Theory of Classifier Combination: The Neural Network Approach. Ph.D Thesis, SUNY at Buffalo (1995)

    Google Scholar 

  8. . Bertolami, R., Bunke, H.: Early feature stream integration versus decision level combination in a multiple classifier system for text line recognition. In: Pattern Recognition, 2006. ICPR 2006. 18th International Conference on. Volume 2. (2006) 845-848

    Google Scholar 

  9. Favata, J.: Character model word recognition. In: Fifth International Workshop on Frontiers in Handwriting Recognition, Essex, England (1996) 437-440

    Google Scholar 

  10. Ho, T.K., Hull, J.J., Srihari, S.N.: Decision combination in multiple classifier sys-tems. IEEE Trans. on Pattern Analysis and Machine Intelligence 16(1) (1994) 66-75

    Google Scholar 

  11. Erp, M.V., Vuurpijl, L.G., Schomaker, L.: An Overview and Comparison of Voting Methods for Pattern Recognition. In: Proc. of the 8th International Workshop on Frontiers in Handwriting Recognition (IWFHR-8), Niagara-on-the-Lake, Canada (2002) 195-200

    Chapter  Google Scholar 

  12. Kang, H.J., Kim, J.: A Probabilistic Framework for Combining Multiple Clas-sifiers at Abstract Level. In: Fourth International Conference on Document Analysis and Recognition (ICDAR), Ulm, Germany (1997) 870-874

    Chapter  Google Scholar 

  13. . Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Trans. on Pattern Analysis and Machine Intelligence (1998) 226-239

    Google Scholar 

  14. Tulyakov, S., Govindaraju, V.: Classifier combination types for biometric appli-cations. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), Workshop on Biometrics, New York, USA (2006)

    Google Scholar 

  15. . Jain, A., Ross, A.: Learning user-specific parameters in a multibiometric system. In: International Conference on Image Processing. 2002. Volume 1. (2002) I-57-I-60 vol.1

    Google Scholar 

  16. Fierrez-Aguilar, J., Garcia-Romero, D., Ortega-Garcia, J., Gonzalez-Rodriguez, J.: Bayesian adaptation for user-dependent multimodal biometric authentication. Pattern Recognition 38(8) (2005) 1317-1319

    Article  Google Scholar 

  17. Auckenthaler, R., Carey, M., Lloyd-Thomas, H.: Score normalization for text-independent speaker verification systems. Digital Signal Processing 10(1-3) (2000) 42-54

    Article  Google Scholar 

  18. . Tulyakov, S., Govindaraju, V.: Identification model for classifier combinations. In: Biometrics Consortium Conference, Baltimore, MD (2006)

    Google Scholar 

  19. Last, M., Bunke, H., Kandel, A.: A feature-based serial approach to classifier combination. Pattern Analysis and Applications 5(4) (2002) 385-398

    Article  MathSciNet  Google Scholar 

  20. Ho, T.K.: The random subspace method for constructing decision forests. Pat-tern Analysis and Machine Intelligence, IEEE Transactions on 20(8) (1998) 832-844

    Google Scholar 

  21. Kleinberg, E.M.: On the algorithmic implementation of stochastic discrimina-tion. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(5) (2000) 473-490

    Article  Google Scholar 

  22. Breiman, L.: Bagging predictors. Machine Learning 24(2) (1996) 123-140

    MATH  MathSciNet  Google Scholar 

  23. Schapire, R.: The strength of weak learnability. Machine Learning 5 (1990) 197-227

    Google Scholar 

  24. Freund, Y., Schapire, R.: Experiments with a New Boosting Algorithm. In: Proc. of 13th Int. Conf. on Machine Learning, Bari, Italy (1996) 148-156

    Google Scholar 

  25. Wolpert, D.H.: Stacked generalization. Neural Networks 5(2) (1992) 241-260

    Article  Google Scholar 

  26. Vapnik, V.: The Nature of Statistical Learning Theory. Springer-Verlag, New York (1995)

    MATH  Google Scholar 

  27. Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)

    MATH  Google Scholar 

  28. Schapire, R., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics 26(5)(1998) 1651-1686

    MATH  MathSciNet  Google Scholar 

  29. . Kleinberg, E.M.: Stochastic discrimination. Annals of Mathematics and Artifi- cial Intelligence 1 (1990)

    Google Scholar 

  30. . Kong, E., Dietterich, T.: Error-correcting output coding corrects bias and vari-ance. In: 12th International Conference on Machine Learning. (1995) 313-321

    Google Scholar 

  31. . Tibshirani, R.: Bias, variance and prediction error for classification rules. Tech-nical Report 41, Department of Statistics, University of Toronto (1996)

    Google Scholar 

  32. Tumer, K., Ghosh, J.: Linear and order statistics combiners for pattern classifi-cation. In Sharkey, A.J., ed.: Combining Artificial Neural Nets: Ensembles and Multi-Net Systems. Springer-Verlag, London (1999) 127-162

    Google Scholar 

  33. Kuncheva, L.: A theoretical study on six classifier fusion strategies. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24(2) (2002) 281-286

    Article  Google Scholar 

  34. . Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley InterScience (2004)

    Google Scholar 

  35. . Fumera, G., Roli, F.: Performance analysis and comparison of linear combiners for classifier fusion. In: SSPR/SPR. (2002) 424-432

    Google Scholar 

  36. Fumera, G., Roli, F.: Analysis of error-reject trade-off in linearly combined multiple classifiers. Pattern Recognition 37(6) (2004) 1245-1265

    Article  MATH  Google Scholar 

  37. Alkoot, F.M., Kittler, J.: Experimental evaluation of expert fusion strategies. Pattern Recognition Letters 20(11-13) (1999) 1361-1369

    Article  Google Scholar 

  38. Freund, Y., Schapire, R.: A Short Introduction to Boosting. Journal of Japanese Society for Artificial Intelligence 14(5) (1999) 771-780

    Google Scholar 

  39. Guenter, S., Bunke, H.: New Boosting Algorithms for Classification Problems with Large Number of Classes Applied to a Handwritten Word Recognition Task. In: 4th International Workshop on Multiple Classifier Systems (MCS), Guildford, UK, Lecture Notes in Computer Science, Springer-Verlag (2003) 326-335

    Chapter  Google Scholar 

  40. Huber, P.J.: Robust Statistics. Wiley, New York (1981)

    Book  MATH  Google Scholar 

  41. Jain, A., Nandakumar, K., Ross, A.: Score Normalization in Multimodal Bio-metric Systems. Pattern Recognition 38(12) (2005) 2270-2285

    Article  Google Scholar 

  42. . Cappelli, R., Maio, D., Maltoni, D.: Combining Fingerprint Classifiers. In: First Int. Workshop on Multiple Classifier Systems. (2000) 351-361

    Google Scholar 

  43. Hampel, F.R., Rousseeuw, P.J., Ronchetti, E., Stahel, W.: Robust Statistics: The Approach Based on Influence Functions. Wiley, New York (1986)

    MATH  Google Scholar 

  44. Velek, O., Jaeger, S., Nakagawa, M.: A New Warping Technique for Normalizing Likelihood of Multiple Classifiers and its Effectiveness in Combined On-Line/ Off-Line Japanese Character Recognition. In: 8th International Workshop on Frontiers in Handwriting Recognition (IWFHR), Niagara-on-the-Lake, Canada (2002) 177-182

    Chapter  Google Scholar 

  45. Velek, O., Jaeger, S., Nakagawa, M.: Accumulated-Recognition-Rate Normaliza-tion for Combining Multiple On/Off-line Japanese Character Classifiers Tested on a Large Database. In: 4th International Workshop on Multiple Classifier Systems (MCS), Guildford, UK, Lecture Notes in Computer Science, Springer-Verlag (2003) 196-205

    Chapter  Google Scholar 

  46. . Jaeger, S.: Informational Classifier Fusion. In: Proc. of the 17th Int. Conf. on Pattern Recognition, Cambridge, UK (2004) 216-219

    Google Scholar 

  47. Jaeger, S.: Using Informational Confidence Values for Classifier Combination: An Experiment with Combined On-Line/Off-Line Japanese Character Recogni-tion. In: Proc. of the 9th Int. Workshop on Frontiers in Handwriting Recognition, Tokyo, Japan (2004) 87-92

    Chapter  Google Scholar 

  48. Jaeger, S., Ma, H., Doermann, D.: Identifying Script on Word-Level with In-formational Confidence. In: Int. Conf. on Document Analysis and Recognition (ICDAR), Seoul, Korea (2005) 416-420

    Chapter  Google Scholar 

  49. Shannon, C.E.: A Mathematical Theory of Communication. Bell System Tech. J. 27(623-656) (1948) 379-423

    MATH  MathSciNet  Google Scholar 

  50. Dempster, A.P.: A Generalization of Bayesian Inference. Journal of the Royal Statistical Society 30 (1968) 205-247

    MathSciNet  Google Scholar 

  51. . Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press (1976)

    Google Scholar 

  52. . Mandler, E., Schuermann, J.: Combining the Classification Results of Inde-pendent Classifiers Based on the Dempster/Shafer Theory of Evidence. In E.S. Gelsema, L.K., ed.: Pattern Recognition and Artificial Intelligence. (1988) 381-393

    Google Scholar 

  53. Huang, Y., Suen, C.: A Method of Combining Multiple Experts for Recognition of Unconstrained Handwritten Numerals. IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI) 17(1) (1995) 90-94

    Article  Google Scholar 

  54. Ianakiev, K., Govindaraju, V.: Architecture for Classifier Combination Us-ing Entropy Measures. In: 1st International Workshop on Multiple Classifier Systems (MCS), Cagliari, Italy, Lecture Notes in Computer Science, Springer-Verlag (2000) 340-350

    Chapter  Google Scholar 

  55. . Ianakiev, K., Govindaraju, V.: Deriving pseudo-probabilities of correctness given scores. In Chen, D., Cheng, X., eds.: Pattern Recognition and String Matching,. Kluwer Publishers (2002)

    Google Scholar 

  56. Woods, K., Kegelmeyer, W.P.J., Bowyer, K.: Combination of multiple classi-fiers using local accuracy estimates. Pattern Analysis and Machine Intelligence, IEEE Transactions on 19(4) (1997) 405-410

    Article  Google Scholar 

  57. Giacinto, G., Roli, F.: Dynamic classifier selection based on multiple classifier behaviour. Pattern Recognition 34(9) (2001) 1879-1881 TY - JOUR.

    Article  MATH  Google Scholar 

  58. . Kuncheva, L.: Clustering-and-selection model for classifier combination. In: Knowledge-Based Intelligent Engineering Systems and Allied Technologies, 2000. Proceedings. Fourth International Conference on. Volume 1. (2000) 185-188 vol. 1

    Google Scholar 

  59. Govindaraju, V., Slavik, P., Xue, H.: Use of lexicon density in evaluating word recognizers. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24(6)(2002) 789-800

    Article  Google Scholar 

  60. . Colombi, J., Reider, J., Campbell, J.: Allowing good impostors to test. In: Signals, Systems & Computers, 1997. Conference Record of the Thirty-First Asilomar Conference on. Volume 1. (1997) 296-300 vol. 1

    Google Scholar 

  61. . Rosenberg, A., Parthasarathy, S.: Speaker background models for connected digit password speaker verification. In: Acoustics, Speech, and Signal Process-ing, 1996. ICASSP-96. Conference Proceedings., 1996 IEEE International Con-ference on. Volume 1. (1996) 81-84 vol. 1

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Unified Biometrics and Sensors, University at Buffalo, Amherst, NY 14228, USA

    Sergey Tulyakov & Venu Govindaraju

  2. Institute for Advanced Computer Studies, University of Maryland, College Park, MD 20742, USA

    Stefan Jaeger & David Doermann

  3. Dept.of Computer Science and Engineering, University at Buffalo, 520 Lee Entrance, Suite 202, UB Commons, Amherst, NY 14228-2567

    Sergey Tulyakov & Venu Govindaraju

  4. Laboratory for Language and Media Processing Institute for Advanced Computer Studies 3451 AV Williams Building, University of Maryland, College Park, Maryland, 20742

    David Doermann

Authors
  1. Sergey Tulyakov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Jaeger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Venu Govindaraju
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Doermann
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dipartimento di Sistemi e Informatica, University of Florence, Via S. Marta, 3, 50139, Firenze, Italy

    Simone Marinai

  2. Hitachi Central Research Laboratory, 1-280, Higashi-Koigakubo, Kokubunji-shi, Tokyo, 185-8601, Japan

    Hiromichi Fujisawa

Rights and permissions

Reprints and permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Tulyakov, S., Jaeger, S., Govindaraju, V., Doermann, D. (2008). Review of Classifier Combination Methods. In: Marinai, S., Fujisawa, H. (eds) Machine Learning in Document Analysis and Recognition. Studies in Computational Intelligence, vol 90. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-76280-5_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-76280-5_14

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-76279-9

  • Online ISBN: 978-3-540-76280-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation