Skip to main content
SpringerLink
Log in

LocalBoost: A Parallelizable Approach to Boosting Classifiers

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Ensemble learning is an active field of research with applications to a broad range of problems. Adaboost is a widely used ensemble approach, however, its computational burden is high because it uses an explicit diversity method for building the individual learners. To address this issue, we present a variant of Adaboost where the learners can be trained in parallel, exchanging information on a sparse collaborative communication that restricts the visibility among them. Experiments on 12 UCI datasets show that this approach is competitive in terms of generalization error but more efficient than Adaboost and two other parallel approximations of this algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Under a client/server (master/slave) model, this cost can become linear if all nodes send their classification to one coordinator that then computes the weight updates and send those weights back to every node. This approach exhibits however more limited scalability because of the synchronization operations and the communication bottleneck around the coordinator [20].

References

  1. Bauer E, Kohavi R (1999) An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn 36:105–139

    Article  Google Scholar 

  2. Bi Y (2012) The impact of diversity on the accuracy of evidential classifier ensembles. Int J Approx Reason 53(4):584–607

    Article  MathSciNet  Google Scholar 

  3. Bradley JK, Schapire RE (2007) Filterboost: regression and classification on large datasets. In: Platt JC, Koller D, Singer Y, Roweis ST (eds) NIPS. Curran Associates, Inc., Red Hook, pp 185–192

    Google Scholar 

  4. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MATH  Google Scholar 

  5. Breiman L (2001) Using iterated bagging to debias regressions. Mach Learn 45(3):261–277

    Article  MATH  Google Scholar 

  6. Brown G, Wyatt JL, Harris R, Yao X (2005) Diversity creation methods: a survey and categorisation. Inf Fusion 6(1):5–20

    Article  Google Scholar 

  7. Brown G, Wyatt JL, Tiňo P (2005) Managing diversity in regression ensembles. J Mach Learn Res 6:1621–1650

    MathSciNet  MATH  Google Scholar 

  8. Bühlmann P (2003) Bagging subagging and bragging for improving some prediction algorithms. In: Akritas MG, Politis DN (eds) Recent advances and trends in nonparametric statistics, Elsevier, New York, pp 19–34

  9. Bukhtoyarov V, Semenkin E (2012) Neural networks ensemble approach for detecting attacks in computer networks. In: 2012 IEEE Congress on evolutionary computation (CEC), pp 1–6

  10. Chen T, Guestrin C (2016) Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 785–794

  11. Chen T, He T (2015) Higgs boson discovery with boosted trees. In: NIPS 2014 workshop on high-energy physics and machine learning, pp 69–80

  12. Deveci M, Rajamanickam S, Leung VJ, Pedretti K, Olivier SL, Bunde DP, Catalyurek UV, Devine K (2014) Exploiting geometric partitioning in task mapping for parallel computers. In: 2014 IEEE 28th international on parallel and distributed processing symposium. IEEE, pp 27–36

  13. Escudero G, Màrquez L, Rigau G (2001) Using lazyboosting for word sense disambiguation. In: The proceedings of the second international workshop on evaluating word sense disambiguation systems, SENSEVAL ’01. Association for Computational Linguistics, Stroudsburg, PA, USA, pp 71–74

  14. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139

    Article  MathSciNet  MATH  Google Scholar 

  15. Galtier V, Genaud S, Vialle S (2009) Implementation of the Adaboost algorithm for large scale distributed environments: comparing JavaSpace and MPJ. In: 2013 international conference on parallel and distributed systems, pp 655–662

  16. Grandvalet Y (2004) Bagging equalizes influence. Mach Learn 55(3):251–270

    Article  MATH  Google Scholar 

  17. Hoefler T, Snir M (2011) Generic topology mapping strategies for large-scale parallel architectures. In: Proceedings of the international conference on supercomputing. ACM, pp 75–84

  18. Kuncheva LI (2004) Combining pattern classifiers: methods and algorithms. Wiley, New York

    Book  MATH  Google Scholar 

  19. Dua D, Karra Taniskidou E (2017) UCI machine learning repository. School of Information and Computer Science, University of California, Irvine, CA. http://archive.ics.uci.edu/ml

  20. Lua EK, Crowcroft J, Pias M, Sharma R, Lim S (2005) A survey and comparison of peer-to-peer overlay network schemes. IEEE Commun Surv Tutor 7(2):72–93

    Article  Google Scholar 

  21. Merler S, Caprile B, Furlanello C (2007) Parallelizing Adaboost by weights dynamics. Comput Stat Data Anal 51(5):2487–2498

    Article  MathSciNet  MATH  Google Scholar 

  22. Mukherjee I, Rudin C, Schapire RE (2013) The rate of convergence of Adaboost. J Mach Learn Res 14:2315–2347

    MathSciNet  MATH  Google Scholar 

  23. \(\tilde{\rm N}\)anculef R, Valle C, Allende H, Moraga C (2012) Training regression ensembles by sequential target correction and resampling. Inf Sci 195:154–174. https://doi.org/10.1016/j.ins.2012.01.035

  24. Opitz D, Maclin R (1999) Popular ensemble methods: an empirical study. J Artif Intell Res 11:169–198

    Article  MATH  Google Scholar 

  25. Palit I, Reddy CK (2012) Scalable and parallel boosting with MapReduce. IEEE Trans Knowl Data Eng 24(10):1904–1916

    Article  Google Scholar 

  26. Poggio T, Rifkin R, Mukherjee S, Rakhlin A (2002) Bagging regularizes. Technical report, AI Memo 2002-003, CBCL Memo 214, MIT AI Lab

  27. Polikar R (2009) Ensemble learning. Scholarpedia 4(1):2776

    Article  Google Scholar 

  28. Ren Y, Zhang L, Suganthan PN (2016) Ensemble classification and regression—recent developments, applications and future directions. IEEE Comput Intell Mag 11(1):41–53

    Article  Google Scholar 

  29. Rudin C, Schapire R, Daubechies I (2007) Precise statements of convergence for Adaboost and ARC-GV. Contemp Math 443:131–146

    Article  MathSciNet  MATH  Google Scholar 

  30. Sluban B, Lavra N (2015) Relating ensemble diversity and performance: a study in class noise detection. Neurocomputing 160:120–131

    Article  Google Scholar 

  31. Tang EK, Suganthan PN, Yao X (2006) An analysis of diversity measures. Mach Learn 65(1):247–271

    Article  Google Scholar 

  32. Valle C, Ñanculef R, Allende H, Moraga C (2007) Two bagging algorithms with coupled learners to encourage diversity. In: IDA, Lecture Notes in Computer Science, vol 4723. Springer, pp 130–139

  33. Valle C, Saravia F, Allende H, Monge R, Fernández C (2010) Parallel approach for ensemble learning with locally coupled neural networks. Neural Process Lett 32(3):277–291

    Article  Google Scholar 

  34. Wu G, Li H, Hu X, Bi Y, Zhang J, Wu X (2009) Mrec4.5: C4.5 ensemble classification with MapReduce. In: 2009 fourth China grid annual conference, pp 249–255

  35. Wu Y, Arribas J (2003) Fusing output information in neural networks: ensemble performs better. In: Proceedings of the 25th annual international conference of the IEEE, vol 3. Engineering in Medicine and Biology Society, pp 2265–2268

  36. Zeng K, Tang Y, Liu F (2011) Parallization of Adaboost algorithm through hybrid MPI/OpenMP and transactional memory. In: Cotronis Y, Danelutto M, Papadopoulos GA (eds) Proceedings of the 19th international Euromicro conference on parallel, distributed and network-based processing, PDP 2011, Ayia Napa, Cyprus, 9–11 Feb 2011. IEEE Computer Society, pp 94–100

  37. Zhang L, Suganthan PN (2014) Random forests with ensemble of feature spaces. Pattern Recognit 47(10):3429–3437

    Article  Google Scholar 

  38. Zhang L, Suganthan PN (2015) Oblique decision tree ensemble via multisurface proximal support vector machine. IEEE Trans Cybern 45(10):2165–2176

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Research Project DGIP-UTFSM (Chile) 116.24.2, Basal Project FB 0821. and from CONICYT Chile through FONDECYT Project 11130122 and FONDECYT Project 1170123.

Author information

Authors and Affiliations

  1. Department of Informatics, Federico Santa María Technical University, Valparaíso, Chile

    Carlos Valle, Ricardo Ñanculef & Héctor Allende

  2. Department of Computing and Informatics, Universidad de Playa Ancha, Valparaíso, Chile

    Carlos Valle

  3. Dortmund University, 44221, Dortmund, Germany

    Claudio Moraga

Authors
  1. Carlos Valle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ricardo Ñanculef
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Héctor Allende
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Claudio Moraga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlos Valle.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Valle, C., Ñanculef, R., Allende, H. et al. LocalBoost: A Parallelizable Approach to Boosting Classifiers. Neural Process Lett 50, 19–41 (2019). https://doi.org/10.1007/s11063-018-9924-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-018-9924-3

Keywords

Search

Navigation