Abstract
Boosting algorithms, as a class of ensemble learning methods, have become very popular in data classification, owing to their strong theoretical guarantees and outstanding prediction performance. However, most of these boosting algorithms were designed for static data, thus they can not be directly applied to on-line learning and incremental learning. In this paper, we propose a novel algorithm that incrementally updates the classification model built upon gradient boosting decision tree (GBDT), namely iGBDT. The main idea of iGBDT is to incrementally learn a new model but without running GBDT from scratch, when new data is dynamically arriving in batch. We conduct large-scale experiments to validate the effectiveness and efficiency of iGBDT. All the experimental results show that, in terms of model building/updating time, iGBDT obtains significantly better performance than the conventional practice that always runs GBDT from scratch when a new batch of data arrives, while still keeping the same classification accuracy. iGBDT can be used in many applications that require in-time analysis of continuously arriving or real-time user-generated data, such as behaviour targeting, Internet advertising, recommender systems, etc.
Similar content being viewed by others
References
Aggarwal CC (2007) Data streams: models and algorithms, vol 31. Springer, Berlin
Babenko B, Yang MH, Belongie S (2009) A family of online boosting algorithms. In: IEEE 12th international conference on computer vision workshops (ICCV workshops). IEEE, pp 1346–1353
Beygelzimer A, Hazan E, Kale S, Luo H (2015a) Online gradient boosting. In: NIPS, pp 2458–2466
Beygelzimer A, Kale S, Luo H (2015b) Optimal and adaptive algorithms for online boosting. In: ICML
Chapelle O, Chang Y (2011) Yahoo! learning to rank challenge overview. In: JMLR proceedings, pp 1–24
Chen ST, Lin HT, Lu CJ (2012) An online boosting algorithm with theoretical justifications. In: ICML
Dheeru D, Karra Taniskidou E (2017) UCI machine learning repository. http://archive.ics.uci.edu/ml. Accessed 31 Jan 2019
Domingos P, Hulten G (2000) Mining high-speed data streams. In: ACM SIGKDD. ACM, pp 71–80
Freund Y (1995) Boosting a weak learning algorithm by majority. Inf Comput 121(2):256–285
Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55:119–139
Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29:1189–1232
Gaber MM, Zaslavsky A, Krishnaswamy S (2005) Mining data streams: a review. ACM Sigmod Rec 34(2):18–26
Grbovic M, Vucetic S (2011) Tracking concept change with incremental boosting by minimization of the evolving exponential loss. In: PKDD. Springer, Berlin, pp 516–532
Hulten G, Spencer L, Domingos PM (2001) Mining time-changing data streams. In: ACM SIGKDD, pp 97–106
Leistner C, Saffari A, Roth PM, Bischof H (2009) On robustness of on-line boosting: a competitive study. In: IEEE 12th international conference on computer vision workshops (ICCV workshops), pp 1362–1369
Liu X, Yu T (2007) Gradient feature selection for online boosting. In: 2007 IEEE 11th international conference on computer vision (ICCV). IEEE, pp 1–8
Oza NC, Russell S (2001) Experimental comparisons of online and batch versions of bagging and boosting. In: ACM SIGKDD. ACM, pp 359–364
Oza NC, Russell SJ (2001) Online bagging and boosting. In: Eighth international workshop on artificial intelligence and statistics, pp 105–112
Pavlov DY, Gorodilov A, Brunk CA (2010) Bagboo: a scalable hybrid bagging-the-boosting model. In: Proceedings of the 19th ACM conference on information and knowledge management, CIKM 2010, Toronto, Ontario, Canada, October 26–30, 2010, pp 1897–1900
Pelossof R, Jones M, Vovsha I, Rudin C (2009) Online coordinate boosting. In: IEEE 12th international conference on computer vision workshops (ICCV workshops). IEEE, pp 1354–1361
Perkins S, Lacker K, Theiler J (2003) Grafting: fast, incremental feature selection by gradient descent in function space. J Mach Learn Res 3(Mar):1333–1356
Zhang C, Hao Y, Mazuran M, Zaniolo C, Mousavi H, Masseglia F (2013) Mining frequent itemsets over tuple-evolving data streams. In: Proceedings of the 28th annual ACM symposium on applied computing, SAC’13, Coimbra, Portugal, March 18–22, 2013, pp 267–274
Zhang C, Liu C, Zhang X, Almpanidis G (2017) An up-to-date comparison of state-of-the-art classification algorithms. Expert Syst Appl 82:128–150. https://doi.org/10.1016/j.eswa.2017.04.003
Zhang T, Yu B (2005) Boosting with early stopping: convergence and consistency. Ann Stat 33:1538–1579
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, C., Zhang, Y., Shi, X. et al. On Incremental Learning for Gradient Boosting Decision Trees. Neural Process Lett 50, 957–987 (2019). https://doi.org/10.1007/s11063-019-09999-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-019-09999-3