Skip to main content
SpringerLink
Log in

Bias vs Variance Decomposition for Regression and Classification

  • Chapter
  • First Online:
Data Mining and Knowledge Discovery Handbook

Summary

In this chapter, the important concepts of bias and variance are introduced. After an intuitive introduction to the bias/variance tradeoff, we discuss the bias/variance decompositions of the mean square error (in the context of regression problems) and of the mean misclassification error (in the context of classification problems). Then, we carry out a small empirical study providing some insight about how the parameters of a learning algorithm influence bias and variance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 349.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Bauer, E., Kohavi, R. An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants. Machine Learning 1999; 36:105-139.

    Article  Google Scholar 

  • Bishop, C.M. Neural Networks for Pattern Recognition. Oxford: Oxford University Press, 1995.

    Google Scholar 

  • Breiman, L., Friedman, J.H., Olsen, R.A., Stone, C.J. Classification and Regression Trees. California: Wadsworth International, 1984.

    MATH  Google Scholar 

  • Breiman, L. Bias, Variance, and Arcing Classifiers. Technical Report 460, Statistics Department, University of California Berkeley, 1996.

    Google Scholar 

  • Breiman, L. Bagging Predictors. Machine Learning 1996; 24(2):123-140.

    MATH  MathSciNet  Google Scholar 

  • Breiman, L. Randomizing Outputs to Increase Prediction Accuracy. Machine Learning 2000; 40(3):229-242.

    Article  MATH  Google Scholar 

  • Dietterich, T.G. and Kong, E.B. Machine Learning Bias, Statistical Bias, and Statistical Variance of Decision Tree Algorithms. Technical Report. Department of Computer Science, Oregon State University, 1995.

    Google Scholar 

  • Domingos, P. An unified Bias-Variance Decomposition for Zero-One and Squa-red Loss. Proceedings of the 17th International Conference on Machine Learning, Morgan Kaufman, San Francisco, CA, 2000.

    Google Scholar 

  • Efron, B., Tibshirani, R.J. An Introduction to the Bootstrap. Chapman & Hall, 1993.

    Google Scholar 

  • Freund, Y., Schapire, R.E. A Decision-Theoretic Generalization of Online Learning and an Application to Boosting. Proceedings of the second European Conference on Computational Learning Theory, 1995.

    Google Scholar 

  • Friedman, J.H. On Bias, Variance, 0/1-Loss, and the Curse-of-Dimensionality. Data Mining and Knowledge Discovery 1997; 1:55-77.

    Article  Google Scholar 

  • Geman, S., Bienenstock, E. and Doursat, R. Neural Networks and the Bias/Vari-ance Dilemna. Neural computation 1992; 4:1-58.

    Article  Google Scholar 

  • Geurts, P. Contribution to Decision Tree Induction: Bias/Variance Tradeoff and Time Series Classification. Phd thesis. Department of Electrical Engineering and Computer Science, University of Liège, 2002.

    Google Scholar 

  • Hansen, J.V. Combining Predictors: Meta Machine Learning Methods and Bias/Variance & Ambiguity Decompositions. PhD thesis. Department of Computer Science, University of Aarhus, 2000.

    Google Scholar 

  • Heskes, T. Bias/Variance Decompositions for Likelihood-Based Estimators. Neural Computation 1998; 10(6):1425-1433.

    Article  Google Scholar 

  • James, G.M. Variance and Bias for General Loss Functions. Machine Learning 2003; 51:115- 135.

    Article  MATH  Google Scholar 

  • Kohavi, R. and Wolpert, D. H. Bias Plus Variance Decomposition for Zero-One Loss Functions. Proceedings of the 13th International Conference on Machine Learning, Morgan Kaufman, 1996.

    Google Scholar 

  • Tibshirani, R. Bias, Variance and Prediction Error for Classification Rules. Technical Report, Department of Statistics, University of Toronto, 1996.

    Google Scholar 

  • Wolpert, D.H. On Bias plus Variance. Neural Computation 1997; 1211-1243.

    Google Scholar 

  • Webb, G. MultiBoosting: A Technique for Combining Boosting and Wagging. Machine Learning 2000; 40(2):159-196.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium

    Pierre Geurts (Postdoctoral Researcher)

  2. F.N.R.S., Liège, Belgium

    Pierre Geurts (Postdoctoral Researcher)

Authors
  1. Pierre Geurts
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. , Dept. Industrial Engineering, Tel Aviv University, Ramat Aviv, 69978, Israel

    Oded Maimon

  2. , Dept. Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva, 84105, Israel

    Lior Rokach

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Geurts, P. (2009). Bias vs Variance Decomposition for Regression and Classification. In: Maimon, O., Rokach, L. (eds) Data Mining and Knowledge Discovery Handbook. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-09823-4_37

Download citation

  • DOI: https://doi.org/10.1007/978-0-387-09823-4_37

  • Published:

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-0-387-09822-7

  • Online ISBN: 978-0-387-09823-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation