Abstract
Is “useful diversity” a myth? Many experiments and the little available theory on diversity in classifier ensembles are either inconclusive, too heavily assumption-bound or openly non-supportive of the intuition that diverse classifiers fare better than non-divers ones. Although a rough general tendency was confirmed in our previous studies, no prominent link appeared between diversity of the ensemble and its accuracy. Diversity alone is a poor predictor of the ensemble accuracy. But there is no agreed definition of diversity to start with! Can we borrow a concept of diversity from biology? How can diversity, as far as we can define and measure it, be used to improve the ensemble? Here we argue that even without a clear-cut definition and theory behind it, studying diversity may prompt viable heuristic solutions. We look into some ways in which diversity can be used in analyzing, selecting or training the ensemble.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Breiman, L.: Random forests. Machine Learning 45, 5–32 (2001)
Cunningham, P., Carney, J.: Diversity versus quality in classification ensembles based on feature selection. Technical Report TCD-CS-2000-02, Department of Computer Science, Trinity College Dublin (2000)
Dietterich, T.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting and randomization. Machine Learning 40(2), 139–157 (2000)
Fleiss, J.L.: Statistical Methods for Rates and Proportions. John Wiley & Sons, Chichester (1981)
Ghosh, J.: Multiclassifier systems: Back to the future. In: Roli, F., Kittler, J. (eds.) MCS 2002. LNCS, vol. 2364, pp. 1–15. Springer, Heidelberg (2002)
Giacinto, G., Roli, F.: Design of effective neural network ensembles for image classification processes. Image Vision and Computing Journal 19(9-10), 699–707 (2001)
Ho, T.K.: The random space method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)
Ho, T.K.: Multiple classifier combination: Lessons and the next steps. In: Kandel, A., Bunke, H. (eds.) Hybrid Methods in Pattern Recognition, pp. 171–198. World Scientific Publishing, Singapore (2002)
Kleinberg, E.M.: Stochastic discrimination. Annals of Mathematics and Artificial Intelligence 1, 207–239 (1990)
Kohavi, R., Wolpert, D.H.: Bias plus variance decomposition for zero-one loss functions. In: Saitta, L. (ed.) Machine Learning: Proc. 13th International Conference, pp. 275–283. Morgan Kaufmann, San Francisco (1996)
Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation and active learning. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems, vol. 7, pp. 231–238. MIT Press, Cambridge (1995)
Kuncheva, L.I.: Fuzzy Classifier Design. Studies in Fuzziness and Soft Computing. Springer, Heidelberg (2000)
Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles. Machine Learning 51, 181–207 (2003)
Lam, L.: Classifier combinations: implementations and theoretical issues. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 77–86. Springer, Heidelberg (2000)
Littlewood, B., Miller, D.R.: Conceptual modeling of coincident failures in multiversion software. IEEE Transactions on Software Engineering 15(12), 1596–1614 (1989)
Margineantu, D.D., Dietterich, T.G.: Pruning adaptive boosting. In: Proc. 14th International Conference on Machine Learning, San Francisco, pp. 378–387. Morgan Kaufmann, San Francisco (1997)
Pękalska, E.z., Duin, R.P.W., Skurichina, M.: A discussion on the classifier projection space for classifier combining. In: Roli, F., Kittler, J. (eds.) MCS 2002. LNCS, vol. 2364, pp. 137–148. Springer, Heidelberg (2002)
Rao, C.R.: Diversity: Its measurement, decomposition, apportionment and analysis. Sankya: The Indian Journal of Statistics, Series A 44(1), 1–22 (1982)
Roli, F., Giacinto, G., Vernazza, G.: Methods for designing multiple classifier systems. In: Kittler, J., Roli, F. (eds.) MCS 2001. LNCS, vol. 2096, pp. 78–87. Springer, Heidelberg (2001)
Rosen, B.E.: Ensemble learning using decorrelated neural networks. Connection Science 8(3/4), 373–383 (1996)
Shipp, C.A., Kuncheva, L.I.: Relationships between combination methods and measures of diversity in combining classifiers. Information Fusion 3(2), 135–148 (2002)
Skalak, D.B.: The sources of increased accuracy for two proposed boosting algorithms. In: Proc. American Association for Artificial Intelligence, AAAI 1996, Integrating Multiple Learned Models Workshop (1996)
Sneath, P.H.A., Sokal, R.R.: Numerical Taxonomy. W.H. Freeman & Co, New York (1973)
Tumer, K., Ghosh, J.: Linear and order statistics combiners for pattern classification. In: Sharkey, A.J.C. (ed.) Combining Artificial Neural Nets, pp. 127–161. Springer, London (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kuncheva, L.I. (2003). That Elusive Diversity in Classifier Ensembles. In: Perales, F.J., Campilho, A.J.C., de la Blanca, N.P., Sanfeliu, A. (eds) Pattern Recognition and Image Analysis. IbPRIA 2003. Lecture Notes in Computer Science, vol 2652. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-44871-6_130
Download citation
DOI: https://doi.org/10.1007/978-3-540-44871-6_130
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40217-6
Online ISBN: 978-3-540-44871-6
eBook Packages: Springer Book Archive