Abstract
Ensemble pruning is a technique to increase ensemble accuracy and reduce its size by choosing an optimal or suboptimal subset of ensemble members to form subensembles for prediction. A number of greedy ensemble pruning methods that are based on greedy search policy have recently been proposed. In this paper, we contribute a new greedy ensemble pruning method, called EPR, based on replacement policy. Unlike traditional pruning methods, EPR searches for the optimal or suboptimal subensemble with predefined size by iteratively replacing the least important classifier in it with current classifier. Especially, replacement would not occur if the current classifier was the least important one. Also, we adopt diversity measure [1] to theoretically analyze the properties of EPR, based on which a new metric is proposed to guide EPR’s search process. We evaluate the performance of EPR by comparing it with other advanced greedy ensemble pruning methods and obtain very promising results.
Supported by the National Science Foundation of China (No. 60901078).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ho, T.K.: The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)
Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. John Wiley and Sons, Chichester (2004)
Breiman, L.: Bagging predictors. Machine Learning 24, 123–140 (1996)
Freund, Y., Schapire, R.F.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997)
Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)
RodrÃguez, J.J., Kuncheva, L.I., Alonso, C.J.: Rotation forest: A new classifier ensemble method. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(10), 1619–1630 (2006)
Zhang, D., Chen, S., Zhou, Z., Yang, Q.: Constraint projections for ensemble learning. In: Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (AAAI 2008), pp. 758–763 (2008)
Zhou, Z.H., Wu, J., Tang, W.: Ensembling neural networks: Many could be better than all. Artificial Intelligence 137(1-2), 239–263 (2002)
Zhang, Y., Burer, S., Street, W.N.: Ensemble pruning via semi-definite programming. Journal of Machine Learning Research 7, 1315–1338 (2006)
Margineantu, D.D., Dietterich, T.G.: Pruning adaptive boosting. In: Proceedings of the 14th International Conference on Machine Learning, pp. 211–218 (1997)
Tamon, C., Xiang, J.: On the Boosting Pruning Problem. In: Lopez de Mantaras, R., Plaza, E. (eds.) ECML 2000. LNCS (LNAI), vol. 1810, pp. 404–412. Springer, Heidelberg (2000)
Fan, W., Chun, F., Wang, H.X., Yu, P.S.: Pruning and dynamic scheduling of cost-sensitive ensembles. In: Proceeding of Eighteenth National Conference on Artificial intelligence, AAAI, pp. 145–151 (2002)
Caruana, R., Niculescu-Mizil, A., Crew, G., Ksikes, A.: Ensemble Selection from Librariries of Models. In: Proceedings of the Twenty-First International Conference (2004)
Martinez-Muverbnoz, G., Suarez, A.: Aggregation ordering in bagging. In: Proceeding of The International Conference on Artificial Intelligence and Applications (IASTED), pp. 258–263. Acta press, Calgary (2004)
Martinez-Muverbnoz, G., Suarez, A.: Pruning in ordered bagging ensembles. In: Proceeding of the 23rd International Conference on Machine Learning, pp. 609–616 (2006)
Lu, Z.Y., Wu, X.D., Zhu, X.Q., Bongard, J.: Ensemble Pruning via Individual Contribution Ordering. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 871–880 (2010)
Banfield, R.E., Hall, L.O., Bowyer, K.W., Kegelmeyer, W.P.: Ensemble diversity measures and their application to thinning. Information Fusion 6(1), 49–62 (2005)
Partalas, I., Tsoumakas, G., Vlahavas, I.P.: Focused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection. In: 18th European Conference on Artificial Intelligence, pp. 117–121 (2008)
Partalas, I., Tsoumakas, G., Vlahavas, I.P.: An ensemble uncertainty aware measure for directed hill climbing ensemble pruning. Machine Learning, 257–282 (2010)
Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensemble and their relationship with the ensemble accuracy. Machine Learning 15(2), 181–207 (2003)
Asuncion, D.N.A.: UCI machine learning repository (2007)
Quinlan, J.R.: C4.5: programs for machine learning. Morgan Kaufmann, San Francisco (1993)
Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)
Demsar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Guo, H., Fan, M. (2011). Ensemble Pruning via Base-Classifier Replacement. In: Wang, H., Li, S., Oyama, S., Hu, X., Qian, T. (eds) Web-Age Information Management. WAIM 2011. Lecture Notes in Computer Science, vol 6897. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23535-1_43
Download citation
DOI: https://doi.org/10.1007/978-3-642-23535-1_43
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23534-4
Online ISBN: 978-3-642-23535-1
eBook Packages: Computer ScienceComputer Science (R0)