Abstract
In this article we develop a class of stochastic boosting (SB) algorithms, which build upon the work of Holmes and Pintore (Bayesian Stat. 8, Oxford University Press, Oxford, 2007). They introduce boosting algorithms which correspond to standard boosting (e.g. Bühlmann and Hothorn, Stat. Sci. 22:477–505, 2007) except that the optimization algorithms are randomized; this idea is placed within a Bayesian framework. We show that the inferential procedure in Holmes and Pintore (Bayesian Stat. 8, Oxford University Press, Oxford, 2007) is incorrect and further develop interpretational, computational and theoretical results which allow one to assess SB’s potential for classification and regression problems. To use SB, sequential Monte Carlo (SMC) methods are applied. As a result, it is found that SB can provide better predictions for classification problems than the corresponding boosting algorithm. A theoretical result is also given, which shows that the predictions of SB are not significantly worse than boosting, when the latter provides the best prediction. We also investigate the method on a real case study from machine learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Andrieu, C., Doucet, A., Holenstein, R.: Particle Markov chain Monte Carlo methods (with discussion). J. R. Stat. Soc. B 72, 1–33 (2010)
Breiman, L.: Arcing classifiers (with discussion). Ann. Stat. 26, 801–849 (1998)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Bühlmann, P., Hothorn, T.: Boosting algorithms: Regularization, prediction and model fitting. Stat. Sci. 22, 477–505 (2007)
Bühlmann, P., Yu, B.: Boosting with the \(\mathbb{L}_{2}\) loss: regression and classification. J. Am. Stat. Ass 98, 324–339 (2003)
Chipman, H., George, E., McCulloch, R.E.: BART: Bayesian additive regression trees. Technical Report, Wharton School (2008)
Del Moral, P.: Feynman-Kac Formulae. Genealogical and Interacting Particle Systems. Springer, New York (2004)
Del Moral, P., Doucet, A., Jasra, A.: Sequential Monte Carlo samplers. J. R. Stat. Soc. B 68, 411–436 (2006)
Doucet, A., Godsill, S., Andrieu, C.: On sequential Monte Carlo sampling methods for Bayesian filtering. Stat. Comput. 10, 197–208 (2000)
Doucet, A., De Freitas, J.F.G., Gordon, N.J.: Sequential Monte Carlo Methods in Practice. Springer, New York (2001)
Doucet, A., Briers, M., Sénécal, S.: Efficient block sampling strategies for sequential Monte Carlo methods. J. Comput. Graph. Stat. 15, 693–711 (2006)
Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. In: Proc. 2nd Europ. Conf. Computat. Learn. Theory. Springer, Berlin (1995)
Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: Proc. 13th Intern. Conf. Mach. Learn. Morgan Kaufmann, San Francisco (1996)
Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 119–139 (1997)
Freund, Y., Mansour, Y., Schapire, R.E.: Generalization bounds for averaged classifiers. Ann. Stat. 32, 1698–1722 (2004)
Friedman, J.: Stochastic gradient boosting. Comput. Stat. Data Anal. 38, 367–378 (2002)
Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting (with discussion). Ann. Stat. 28, 337–407 (2000)
Gneiting, T., Balabdaoui, F., Raftery, A.E.: Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. B 69, 243–268 (2007)
Green, P.J.: Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82, 711–732 (1995)
Hibon, M., Evgeniou, T.: To combine or not to combine: selecting among forecasts and their combinations. Int. J. Forecast. 21, 15–24 (2005)
Holmes, C.C., Pintore, A.: Bayesian relaxation: Boosting, the lasso and other L α -norms (with discussion). In: Bayarri, S., Berger, J.O., Bernardo, J.M., Dawid, A.P., Heckerman, D., Smith, A.F.M., West, M. (eds.) Bayesian Stat. 8, pp. 253–282. Oxford University Press, Oxford (2007)
Jiang, W.: Process consistency for AdaBoost. Ann. Stat. 32, 13–29 (2004)
Mease, D., Wyner, A.: Evidence contrary to the statistical view of boosting (with discussion). J. Mach. Learn. Res. 9, 131–201 (2008)
Nakada, Y., Mouri, Y., Hongo, Y., Matsumoto, T.: Gibbsboost: a boosting algorithm using a sequential Monte Carlo approach. In: Proc. IEEE Sig. Proc. Work (2006)
Nobile, A.: Bayesian Analysis of Finite Mixture Distributions. Ph.D. Thesis, Carnegie Mellon University (1994)
Snoek, G.M., Worring, M., van Gemert, J.C., Geusebroek, J.M., Smeudlders, A.W.M.: The challenge problem for automated detection of 101 semantic concepts in multimedia. In: Proc. ACM Multimedia (2006)
Zhang, T., Yu, B.: Boosting with early stopping: Convergence and consistency. Ann. Stat. 33, 1518–1579 (2005)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jasra, A., Holmes, C.C. Stochastic boosting algorithms. Stat Comput 21, 335–347 (2011). https://doi.org/10.1007/s11222-010-9173-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11222-010-9173-4