Abstract
It is well known that ensembles of predictors produce better accuracy than a single predictor provided there is diversity in the ensemble. This diversity manifests itself as disagreement or ambiguity among the ensemble members. In this paper we focus on ensembles of classifiers based on different feature subsets and we present a process for producing such ensembles that emphasizes diversity (ambiguity) in the ensemble members. This emphasis on diversity produces ensembles with low generalization errors from ensemble members with comparatively high generalization error. We compare this with ensembles produced focusing only on the error of the ensemble members (without regard to overall diversity) and find that the ensembles based on ambiguity have lower generalization error. Further, we find that the ensemble members produced focusing on ambiguity have less features on average that those based on error only. We suggest that this indicates that these ensemble members are local learners.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Breiman, L., (1996) Bagging predictors. Machine Learning, 24:123–140.
Byrne, S., Cunningham, P., Barry, A., Graham, I., Delaney T., Corrigan, O.I., (2000) Using Neural Nets for Decision Support in Prescription and Outcome Prediction in Anticoagulation Drug Therapy, N. Lavrac, S. Miksch (eds.): The Fifth Workshop on Intelligent Data Analysis in Medicine and Pharmacology (IDAMAP-2000).
Cherkauer, K.J. (1995) Stuffing Mind into Computer: Knowledge and Learning for Intelligent Systems. Informatica 19:4 (501–511) Nov. 1995
Condorcet, Marquis J. A. (1781) Sur les elections par scrutiny, Histoire de l’Academie Royale des Sciences, 31–34.
Cunningham, P., Carney, J., (2000) Diversity versus Quality in Classification Ensembles based on Feature Selection, 11th European Conference on Machine Learning (ECML 2000), Lecture Notes in Artificial Intelligence, R. López de Mántaras and E. Plaza, (eds) pp109–116, Springer Verlag.
Cunningham, P., & Zenobi, G., (2001) Case Representation Issues for Case-Based Reasoning from Ensemble Research, to be presented at ICCBR 2001.
Guerra-Salcedo, C., Whitley, D., (1999a). Genetic Approach for Feature Selection for Ensemble Creation. in GECCO-99: Proceedings of the Genetic and Evolutionary Computation Conference, Banzhaf, W., Daida, J., Eiben, A. E., Garzon, M. H., Honavar, V., Jakiela, M., & Smith, R. E. (eds.). Orlando, Florida USA, pp236–243, San Francisco, CA: Morgan Kaufmann.
Guerra-Salcedo, C., Whitley, D., (1999b). Feature Selection Mechanisms for Ensemble Creation: A Genetic Search Perspective, in Data Mining with Evolutionary Algorithms: Research Directions. Papers from the AAAI Workshop. Alex A. Freitas (Ed.) Technical Report WS-99-06. AAAI Press, 1999.
Hansen, L.K., Salamon, P., (1990) Neural Network Ensembles, IEEE Pattern Analysis and Machine Intelligence, 1990. 12, 10, 993–1001.
Ho, T.K., (1998a) The Random Subspace Method for Constructing Decision Forests, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 8, 832–844.
Ho, T.K., (1998b) Nearest Neighbours in Random Subspaces, Proc. Of 2nd International Workshop on Statistical Techniques in Pattern Recognition, A. Amin, D. Dori, P. Puil, H. Freeman, (eds.) pp640–648, Springer Verlag LNCS 1451.
Kohavi, R. & John, G.H., (1998) The Wrapper Approach, in Feature Selection for Knowledge Discovery and Data Mining, H. Liu & H. Motoda (eds.), Kluwer Academic Publishers, pp33–50.
Krogh, A., Vedelsby, J., (1995) Neural Network Ensembles, Cross Validation and Active Learning, in Advances in Neural Information Processing Systems 7, G. Tesauro, D. S. Touretsky, T. K. Leen, eds., pp231–238, MIT Press, Cambridge MA.
Liu Y., Yao X. (1999) Ensemble learning via negative correlation, Neural Networks 12, 1999.
Nitzan, S.I., Paroush, J., (1985) Collective Decision Making. Cambridge: Cambridge University Press.
Opitz D., Shavlik J., (1996) Generating Accurate and diverse members of a Neural Network Ensemble, Advances in Neural Information Processing Systems, pp. 535–543, Denver, CO. MIT Press. 1996.
Tumer, K., and Ghosh, J., (1996) Error Correlation and Error Reduction in Ensemble Classifiers, Connection Science, Vol. 8, No. 3 & 4, pp 385–404.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zenobi, G., Cunningham, P. (2001). Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error. In: De Raedt, L., Flach, P. (eds) Machine Learning: ECML 2001. ECML 2001. Lecture Notes in Computer Science(), vol 2167. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44795-4_49
Download citation
DOI: https://doi.org/10.1007/3-540-44795-4_49
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42536-6
Online ISBN: 978-3-540-44795-5
eBook Packages: Springer Book Archive