Skip to main content
SpringerLink
Log in

A new accelerated proximal boosting machine with convergence rate \(O(1/t^2)\)

  • Original Research
  • Published:
Journal of Applied Mathematics and Computing Aims and scope Submit manuscript

Abstract

Boosting algorithms, a well-studied and effective machine learning technology, iteratively combines the output of weak learners to produce a powerful predictive model with performances improved over the single weak learner. Accelerated proximal boosting machine (APBM) is a boosting algorithm built upon the proximal point algorithm and employed Nesterov’s accelerated descent. However, in the acceleration process, the momentum term will accumulate errors, which may cause the algorithm to diverge. In this paper, we propose a new algorithm named CAPBM, which incorporates corrected pseudo residual into the design of APBM in boosting process. By applying the corrected pseudo residual to update the momentum sequence, CAPBM can reduce the error generated when fitting the residual, and further ensure the algorithm convergence. Finally, we theoretically prove that CAPBM is convergent with a rate of \(O(1/t^2)\). Numerical experiments on real datasets are carried out to illustrate that the proposed CAPBM algorithm is effective and competitive.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/.

References

  1. Schapire, R.E.: The strength of weak learnability. Mach. Learn. 5(2), 197–227 (1990)

    Article  Google Scholar 

  2. Freund, Y.: Boosting a weak learning algorithm by majority. Inf. Comput. 121(2), 256–285 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  3. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. ICML 96, 148–156 (1996)

    Google Scholar 

  4. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  5. Breiman, L.: Arcing the edge, Technical report, Technical Report 486. University of California, Statistics Department (1997)

  6. Breiman, L.: Arcing classifier (with discussion and a rejoinder by the author). Ann. Stat. 26(3), 801–849 (1998)

    Article  MATH  Google Scholar 

  7. Breiman, L.: Prediction games and arcing algorithms. Neural Comput. 11(7), 1493–1517 (1999)

    Article  Google Scholar 

  8. Breiman, L.: Some infinity theory for predictor ensembles, Technical Report, Technical Report 579. Statistics Dept, UCB (2000)

  9. Breiman, L.: Population theory for boosting ensembles. Ann. Stat. 32(1), 1–11 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  10. Friedman, J.H., Hastie, T., Tibshirani, R., et al.: Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). Ann. Stat. 28(2), 337–407 (2000)

    Article  MATH  Google Scholar 

  11. Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. pp. 1189-1232 (2001)

  12. Friedman, J.H.: Stochastic gradient boosting. Comput. Stat. Data Anal. 38(4), 367–378 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  13. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  14. Meng, X., Bradley, J., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D.B., Amde, M., Owen, S., et al.: Mllib: machine learning in apache spark. J. Mach. Learn. Res. 17(1), 1235–1241 (2016)

    MathSciNet  MATH  Google Scholar 

  15. Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)

  16. Ke, G. Meng, Q. Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.-Y.: Lightgbm: A highly efficient gradient boosting decision tree. Adv. Neural Inf. Process. Syst., 3146–3154 (2017)

  17. Ponomareva, N., Radpour, S., Hendry, G., Haykal, S., Colthurst, T., Mitrichev, P., Grushetsky, A.: Tf boosted trees: a scalable tensorflow based framework for gradient boosting. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, pp. 423–427 (2017)

  18. Wang, K., Wang, Y., Zhao, Q., Meng, D., Liao, X., Zongben, X.: Splboost: an improved robust boosting algorithm based on self-paced learning. IEEE Trans. Cybern. 51(3), 1556–1570 (2019)

    Article  Google Scholar 

  19. Mason, L., Baxter, J., Bartlett, P., Frean, M.: Boosting algorithms as gradient descent. Adv. Neural Inf. Process. Syst. 12, 512–518 (1999)

    Google Scholar 

  20. Collins, M., Schapire, R.E., Singer, Y.: Logistic regression, AdaBoost and Bregman distances. Mach. Learn. 48(1–3), 253–285 (2002)

    Article  MATH  Google Scholar 

  21. Bickel, P.J., Ritov, Y., Zakai, A.: Some theory for generalized boosting algorithms. J. Mach. Learn. Res. 7(5), 705–732 (2006)

    MathSciNet  MATH  Google Scholar 

  22. Telgarsky, M.: A primal-dual convergence analysis of boosting. J. Mach. Learn. Res. 13(1), 561–606 (2012)

    MathSciNet  MATH  Google Scholar 

  23. Haihao, L., Mazumder, R.: Randomized gradient boosting machine. SIAM J. Optim. 30(4), 2780–2808 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  24. Wang, C., Wang, Y., Schapire, R. et al.: Functional Frank–Wolfe boosting for general loss functions. arXiv preprint arXiv:1510.02558 (2015)

  25. Biau, G., Cadre, B., Rouvíére, L.: Accelerated gradient boosting. Mach. Learn. 108(6), 971–992 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  26. Lu, H., Karimireddy, S.P., Ponomareva, N., Mirrokni, V.: Accelerating gradient boosting machines. In: International Conference on Artificial Intelligence and Statistics, pp. 516–526 (2020)

  27. Fouillen, E., Boyer, C., Sangnier, M.: Proximal boosting and its acceleration. arXiv-1808 (2018)

  28. Nesterov, Y.E.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Dordrech (2004)

    Book  MATH  Google Scholar 

  29. Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization. University of Washington, Tech. Rep. (2008)

  30. Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1(3), 127–239 (2014)

    Article  Google Scholar 

  31. Bühlmann, P., Bin, Yu.: Boosting with the l2 loss: regression and classification. J. Am. Stat. Assoc. 98(462), 324–339 (2003)

    Article  MATH  Google Scholar 

  32. Zhang, T., Bin, Yu., et al.: Boosting with early stopping: convergence and consistency. Ann. Stat. 33(4), 1538–1579 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  33. Bartlett, P.L., Traskin, M.: Adaboost is consistent. J. Mach. Learn. Res. 8(10), 2347–2368 (2007)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors are indebted to the editors and anonymous referees for their a number of helpful comments and suggestions that improved the quality of this manuscript.

Author information

Authors and Affiliations

  1. Department of Mathematics, Taiyuan Normal University, Jinzhong, 030619, Shanxi, China

    Jinxiang Li, Fusheng Wang, Na Zhen & Yiming Yang

Authors
  1. Jinxiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fusheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Na Zhen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yiming Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fusheng Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by Research Project Foundation of Shanxi Scholarship Council of China (No.2017-104)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Wang, F., Zhen, N. et al. A new accelerated proximal boosting machine with convergence rate \(O(1/t^2)\). J. Appl. Math. Comput. 68, 3747–3766 (2022). https://doi.org/10.1007/s12190-021-01684-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12190-021-01684-w

Keywords

Mathematics Subject Classification

Search

Navigation