Skip to main content
SpringerLink
Log in

Ensemble selection by GRASP

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Ensemble selection, which aims to select a proper subset of the original whole ensemble, can be seen as a combinatorial optimization problem, and usually can achieve a pruned ensemble with better performance than the original one. Ensemble selection by greedy methods has drawn a lot of attention, and many greedy ensemble selection algorithms have been proposed, many of which focus on the design of a new evaluation measure or on the study about different search directions. It is well accepted that diversity plays a crucial role in ensemble selection methods. Many evaluation measures based on diversity have been proposed and have achieved a good success. However, most of the existing researches have neglected the substantial local optimal problem of greedy methods, which is just the central issue addressed in this paper, where a new Ensemble Selection (GraspEnS) algorithm based on Greedy Randomized Adaptive Search Procedure (GRASP) is proposed. The typical greedy ensemble selection approach is improved by the random factor incorporated into GraspEnS. Moreover, the GraspEnS algorithm realizes multi-start searching and appropriately expands the search range of the typical greedy approaches. Experimental results demonstrate that the newly devised GraspEnS algorithm is able to achieve a final pruned subensemble with comparable or better performance compared with its competitors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 2
Algorithm 4
Algorithm 5
Fig. 3
Algorithm 6
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

References

  1. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55:119–139

    Article  MATH  MathSciNet  Google Scholar 

  2. Quinlan J (1996) Bagging, boosting, and C4.5. Presented at proc. 13th national conference on artificial intelligence, Cambridge, MA

  3. Breiman L (1996) Bagging predictors. Mach Learn 24:123–140

    MATH  MathSciNet  Google Scholar 

  4. Breiman L (1998) Arcing classifiers. Ann Stat 26:801–849

    Article  MATH  MathSciNet  Google Scholar 

  5. Bauer E, Kohavi R (1999) An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn 36:105–139

    Article  Google Scholar 

  6. Sharkey AJC (1999) Combining artificial neural nets: ensemble and modular multi-net systems. Springer, London

    Book  MATH  Google Scholar 

  7. Rätsch G, Onoda T, Müller KR (2001) Soft margins for AdaBoost. Mach Learn 42:287–320

    Article  MATH  Google Scholar 

  8. Martinez-Munoz G, Suarez A (2004) Aggregation ordering in bagging. Presented at international conference on artificial intelligence and applications

  9. Schapire R, Freund Y, Bartlett P, Lee W (1998) Boosting the margin: a new explanation for the effectiveness of voting methods. Ann Stat 12:1651–1686

    MathSciNet  Google Scholar 

  10. Margineantu D, Dietterich T (1997) Pruning adaptive boosting. Presented at proceedings of the 14th international conference on machine learning

  11. Idris A, Khan A, Lee YS (2013) Intelligent churn prediction in telecom: employing mRMR feature selection and RotBoost based ensemble classification. Appl Intell 39:659–672

    Article  Google Scholar 

  12. Nadig K, Potter W, Hoogenboom G, McClendon R (2013) Comparison of individual and combined ANN models for prediction of air and dew point temperature. Appl Intell 39:354–366

    Article  Google Scholar 

  13. Park S, Lee SR (2013) Red tides prediction system using fuzzy reasoning and the ensemble method. Appl Intell. doi:10.1007/s10489-013-0457-1

    Google Scholar 

  14. Caruana R, Munson A, Niculescu-Mizil A (2006) Getting the most out of ensemble selection. Presented at sixth international conference on data mining, Hong Kong

  15. Banfield RE, Hall LO, Bowyer KW, Kegelmeyer WP (2005) Ensemble diversity measures and their application to thinning. Inf Fusion 6:49–62

    Article  Google Scholar 

  16. Li N, Yu Y, Zhou Z-H (2012) Diversity regularized ensemble pruning. In: Machine learning and knowledge discovery in databases. Springer, Berlin, pp 330–345

    Chapter  Google Scholar 

  17. Tsoumakas G, Partalas I, Vlahavas I (2009) An ensemble pruning primer. In: Applications of supervised and unsupervised ensemble methods. Springer, Berlin, pp 1–13

    Chapter  Google Scholar 

  18. Zhenyu L, Xindong W, Xingquan Z, Josh B (2010) Ensemble pruning via individual contribution ordering

  19. Zhang Y, Burer S, Street WN (2006) Ensemble pruning via semi-definite programming. J Mach Learn Res 7:1315–1338

    MATH  MathSciNet  Google Scholar 

  20. Martínez-Muñoz G, Suárez A (2005) Switching class labels to generate classification ensembles. Pattern Recognit 38:1483–1494

    Article  Google Scholar 

  21. Partalas I, Tsoumakas G, Vlahavas I (2009) Pruning an ensemble of classifiers via reinforcement learning. Neurocomputing 72:1900–1909

    Article  Google Scholar 

  22. Zhou Z-H, Wu J, Tang W (2002) Ensembling neural networks: many could be better than all. Artif Intell 137:239–263

    Article  MATH  MathSciNet  Google Scholar 

  23. Martinez-Munoz G, Suarez A (2007) Using boosting to prune bagging ensembles. Pattern Recognit Lett 28:156–165

    Article  Google Scholar 

  24. Tsoumakas G, Angelis L, Vlahavas I (2005) Selective fusion of heterogeneous classifiers. Intell Data Anal 9:511–525

    Google Scholar 

  25. Dai Q (2013) A competitive ensemble pruning approach based on cross-validation technique. Knowl-Based Syst 37:394–414

    Article  Google Scholar 

  26. Dai Q, Liu Z (2013) ModEnPBT: a modified backtracking ensemble pruning algorithm. Appl Soft Comput 13:4292–4302

    Article  Google Scholar 

  27. Caruana R, Niculescu-Mizil A, Crew G, Ksikes A (2004) Ensemble selection from libraries of models. Presented at proceedings of the 21st international conference on machine learning

  28. Zhou Z-H (2012) Ensemble methods: foundations and algorithms. Chapman and Hall/CRC Press, London/Boca Raton

    Google Scholar 

  29. Tsoumakas G, Partalas I, Vlahavas I (2009) Applications of supervised and unsupervised ensemble methods, vol 245. Springer, Berlin

    Book  Google Scholar 

  30. Lee H, Kim E, Pedrycz W (2012) A new selective neural network ensemble with negative correlation. Appl Intell 37:488–498

    Article  Google Scholar 

  31. Tamon C, Xiang J (2000) On the boosting pruning problem. Presented at 11th European conference on machine learning, Springer, Berlin

  32. Partalas I, Tsoumakas G, Vlahavas I (2010) An ensemble uncertainty aware measure for directed hill climbing ensemble pruning. Mach Learn 81:257–282

    Article  MathSciNet  Google Scholar 

  33. Dai Q (2013) A novel ensemble pruning algorithm based on randomized greedy selective strategy and ballot. Neurocomputing 122:258–265

    Article  Google Scholar 

  34. Feo TA, Resende MGC (1995) Greedy randomized adaptive search procedures. J Glob Optim 6:109–133

    Article  MATH  MathSciNet  Google Scholar 

  35. Gevezes T, Pitsoulis L (2013) A greedy randomized adaptive search procedure with path relinking for the shortest superstring problem. J Comb Optim, 1–25

  36. Mousavi SR, Babaie M, Montazerian M (2012) An improved heuristic for the far from most strings problem. J Heuristics 18:239–262

    Article  Google Scholar 

  37. Resende MGC, Ribeiro CC (2003) Greedy randomized adaptive search procedures. In: Glover F, Kochenberger G (eds) Handbook of metaheuristics. Kluwer Academic, Norwell, pp 219–249

    Google Scholar 

  38. Festa P, Resende MGC (2002) GRASP: an annotated bibliography. Kluwer Academic, Norwell

    Google Scholar 

  39. Layeb A, Selmane M, Bencheikh Elhoucine M (2013) A new greedy randomized adaptive search procedure for multiple sequence alignment. Int J Bioinform Res Appl 9:1–14

    Article  Google Scholar 

  40. Fan W, Chu F, Wang H, Yu PS (2002) Pruning and dynamic scheduling of cost-sensitive ensembles. Presented at eighteenth national conference on artificial intelligence, American association for artificial intelligence

  41. Partalas I, Tsoumakas G, Vlahavas I (2012) A study on greedy algorithms for ensemble pruning. Technical report TR-LPIS-360-12, Department of Informatics, Aristotle University of Thessaloniki, Greece

  42. Martinez-Muoz G, Hernández-Lobato D, Suárez A (2009) An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans Pattern Anal Mach Intell 31:245–259

    Article  Google Scholar 

  43. Tang E, Suganthan P, Yao X (2006) An analysis of diversity measures. Mach Learn 65:247–271

    Article  Google Scholar 

  44. Martinez-Munoz G, Hernandez-Lobato D, Suarez A (2009) An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans Pattern Anal Mach Intell 31:245–259

    Article  Google Scholar 

  45. Asuncion A, Newman DJ (2007) UCI machine learning repository

  46. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30

    MATH  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China under the Grant No. 61100108 and Grant No. 61375021, and is supported by the Natural Science Foundation of Jiangsu Province of China under the Grant No. SBK201322136, and is supported by “the Fundamental Research Funds for the Central Universities”, No. NZ2013306, and Qing Lan Project. And we would like to express our appreciation for the valuable comments from reviewers and editors.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Zhuan Liu, Qun Dai & Ningzhong Liu

Authors
  1. Zhuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qun Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ningzhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qun Dai.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liu, Z., Dai, Q. & Liu, N. Ensemble selection by GRASP. Appl Intell 41, 128–144 (2014). https://doi.org/10.1007/s10489-013-0510-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-013-0510-0

Keywords

Search

Navigation