Abstract
By performing experiments on publicly available multi-class datasets we examine the effect of bootstrapping on the bias/variance behaviour of error-correcting output code ensembles. We present evidence to show that the general trend is for bootstrapping to reduce variance but to slightly increase bias error. This generally leads to an improvement in the lowest attainable ensemble error, however this is not always the case and bootstrapping appears to be most useful on datasets where the non-bootstrapped ensemble classifier is prone to overfitting.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bishop, M.C.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)
Breiman, L.: Bagging Predictors. Machine Learning 24(2), 123–140 (1994)
Breiman, L.: Arcing Classifiers. Annals of Statistics 26(3), 801–849 (1998)
Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity Creation Methods: A Survey and Categorisation. Journal of Information Fusion 6(1) (2005)
Burges, C.J.C.: A Tutorial on Support Vector Machines for Pattern Recognition. Knowledge Discovery and Data Mining 2(2) (1998)
Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001), Software: http://www.csie.ntu.edu.tw/~cjlin/libsvm
Dietterich, T.G., Bakiri, G.: Solving Multiclass Learning Problems via Error-Correcting Output Codes. Journal of Artificial Intelligence Research 2, 263–286 (1995)
Dietterich, T.G., Kong, E.B.: Machine learning bias, statistical bias, and statistical variance of decision tree algorithms. Technical Report, Dept. of Computer Science, Oregon State University (1995)
Duin, R.P.W., Juszczak, P., Paclik, P., Pekalska, E., de Ridder, D., Tax, D.M.J., Verzakov, S.: PRTools 4.1, A Matlab Toolbox for Pattern Recognition, Delft University of Technology (2007)
Geman, S., Bienenstock, E.: Neural networks and the bias / variance dilemma. Neural Computation 4, 1–58 (1992)
James, G.: Majority Vote Classifiers: Theory and Applications. Ph.D Dissertation, Stanford University (1998)
James, G.: Variance and Bias for General Loss Functions. Machine Learning 51(2), 115–135 (2003)
Kohavi, R., Wolpert, D.: Bias plus variance decomposition for zero-one loss functions. In: Proc. 13th International Conference on Machine Learning, pp. 275–283 (1996)
Kong, E.B., Dietterich, T.G.: Error-correcting output coding corrects bias and variance. In: Proc. 12th International Conference on Machine Learning, pp. 313–321 (1995)
Merz, C.J., Murphy, P.M.: UCI Repository of Machine Learning Databases (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Valentini, G., Dietterich, T.G.: Bias-Variance Analysis of Support Vector Machines for the Development of SVM-Based Ensemble Methods. Journal of Machine Learning Research 5, 725–775 (2004)
Windeatt, T.: Accuracy/ Diversity and Ensemble Classifier Design. IEEE Trans. Neural Networks 17(4) (July 2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Smith, R.S., Windeatt, T. (2009). The Bias Variance Trade-Off in Bootstrapped Error Correcting Output Code Ensembles. In: Benediktsson, J.A., Kittler, J., Roli, F. (eds) Multiple Classifier Systems. MCS 2009. Lecture Notes in Computer Science, vol 5519. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02326-2_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-02326-2_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02325-5
Online ISBN: 978-3-642-02326-2
eBook Packages: Computer ScienceComputer Science (R0)