Skip to main content
SpringerLink
Log in

An analysis of accuracy-diversity trade-off for hybrid combined system with multiobjective predictor selection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This study examines the contribution of diversity under a multi-objective context for the promotion of learners in an evolutionary system that generates combinations of partially trained learners. The examined system uses a grammar-driven genetic programming to evolve hierarchical, multi-component combinations of multilayer perceptrons and support vector machines for regression. Two advances are studied. First, a ranking formula is developed for the selection probability of the base learners. This formula incorporates both a diversity measure and the performance of learners, and it is tried over a series of artificial and real-world problems. Results show that when the diversity of a learner is incorporated with equal weights to the learner performance in the evolutionary selection process, the system is able to provide statistically significantly better generalization. The second advance examined is a substitution phase for learners that are over-dominated, under a multi-objective Pareto domination assessment scheme. Results here show that the substitution does not improve significantly the system performance, thus the exclusion of very weak learners, is not a compelling task for the examined framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Abbass H (2003) Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization. In: The IEEE Congress on evolutionary computation (IEEE-CEC), vol 3. IEEE Press, Los Alamitos, pp 2074–2080

    Google Scholar 

  2. Alba E, Cotta C, Troya J (1996) Evolutionary design of fuzzy logic controllers using strongly-typed gp. In: Proc 1996 IEEE int’l symposium on intell control, pp 127–132

    Google Scholar 

  3. Andre D, Koza J (1996) Parallel genetic programming: a scalable implementation using the transputer network. In: Advances in genetic programming, pp 317–337

    Google Scholar 

  4. Banfield R, Hall L, Bowyer K, Kegelmeyer W (2005) Ensemble diversity measures and their application to thinning. Inf Fusion 6(1):49–62

    Article  Google Scholar 

  5. Barrero D, R-Moreno M, Castano B, Camacho D (2011) An empirical study on the accuracy of computational effort in genetic programming. In: 2011 IEEE Congress on evolutionary computation (CEC), pp 1164–1171

    Chapter  Google Scholar 

  6. Basseur M, Zitzler E (2006) Handling uncertainty in indicator-based multiobjective optimization. Int J Comput Intell Res 2(3):255–272

    Article  MathSciNet  Google Scholar 

  7. Bentley PJ, Wakefield JP (1997) Finding acceptable solutions in the pareto-optimal range using multiobjective genetic algorithms. In: Soft computing in engineering design and manufacturing, vol 5, pp 231–240

    Google Scholar 

  8. Bishop C (1995) Neural networks for pattern recognition. Oxford University Press, London

    Google Scholar 

  9. Brown G, Wyatt J, Harris R, Yao X (2005) Diversity creation methods: a survey and categorisation. J Inf Fusion 6:5–20

    Article  Google Scholar 

  10. Budka M, Gabrys B (2010) Ridge regression ensemble for toxicity prediction. Proc Comput Sci 1(1):193–201

    Article  Google Scholar 

  11. Chandra A, Yao X (2006) Ensemble learning using multi-objective evolutionary algorithms. J Math Model Algorithms 5(4):417–445

    MATH  MathSciNet  Google Scholar 

  12. Christensen S, Oppacher F (2002) An analysis of Koza’s computational effort statistic for genetic programming. In: Proc EuroGP ’02, 5th European conference on genetic programming. Springer, Berlin, pp 182–191

    Google Scholar 

  13. Clemen R (1989) Combining forecasts: a review and annotated bibliography. Int J Forecast 5(4):559–583

    Article  Google Scholar 

  14. Coelho A, Fernandes E, Faceli K (2011) Multi-objective design of hierarchical consensus functions for clustering ensembles via genetic programming. Decis Support Syst 51(4):794–809

    Article  Google Scholar 

  15. Colobert R, Bengio S (2001) Svmtorch: support vector machines for large- scale regression problems. J Mach Learn Res 1:143–160

    MathSciNet  Google Scholar 

  16. Detrano R, Janosi A, Steinbrunn W, Pfisterer M, Schmid J, Sandhu S, Guppy K, Lee S, Froelicher V (1989) International application of a new probability algorithm for the diagnosis of coronary artery disease. Am J Cardiol 64:304–310

    Article  Google Scholar 

  17. Dietterich T (2000) Ensemble methods in machine learning. Multiple Classif Syst 1557:1–15

    Article  Google Scholar 

  18. Drechsler N, Drechsler R, Becker B (2001) Multi-objective optimisation based on relation favour. In: 1st int’l conf on evol multi-criterion optimization. Springer, Berlin, pp 105–145

    Google Scholar 

  19. Falco I, Tarantino E, Cioppa A, Gagliardi F (2005) A novel grammar-based genetic programming approach to clustering. In: Proc 2005 ACM symp on appl comp, Santa Fe, New Mexico, pp 928–932

    Chapter  Google Scholar 

  20. Fernandez F, Tommassini M, Vanneschi L (2003) An empirical study of multipopulation genetic programming. Genet Program Evol Mach 4(1)

  21. Folino G, Pizzuti C, Spezzano G Ensemble techniques for parallel genetic programming based classifiers

  22. Folino G, Pizzuti C, Spezzano G, Vanneschi L, Tommassini M (2003) Diversity analysis in cellular and multipopulation genetic programming. In: Proc of the 2003 Congress on evolutionary computation, CEC2003

    Google Scholar 

  23. Fonseca C, Fleming P (1993) Genetic algorithms for multiobjective optimization: formulation, discussion and generalization. In: 5th international conference genetic algorithms, pp 416–423

    Google Scholar 

  24. Friedman J (1991) Multivariate adaptive regression splines. Ann Stat 19:1–141

    Article  MATH  Google Scholar 

  25. Gabrys B (2004) Learning hybrid neuro-fuzzy classifier models from data: to combine or not to combine? Fuzzy Sets Syst 147:39–56

    Article  MATH  MathSciNet  Google Scholar 

  26. Gacto MJ, Alcala R, Herrera F (2012) A multi-objective evolutionary algorithm for an effective tuning of fuzzy logic controllers in heating, ventilating and air conditioning systems. Appl Intell 36(2):330–347

    Article  Google Scholar 

  27. Hong JH, Cho SB (2006) The classification of cancer based on DNA microarray data that uses diverse ensemble genetic programming. Artif Intell Med 36(1):43–58

    Article  Google Scholar 

  28. Jacobs R (1997) Bias-variance analyses of mixture-of-experts architectures. Neural Comput 9:369–383

    Article  MATH  Google Scholar 

  29. Jang JS (1993) Anfis: Adaptive-network-based fuzzy inference system. IEEE Trans Syst Man Cyber 23(3)

  30. Jordan MI, Jacobs RA (1993) Hierarchical mixtures of experts and the em algorithm. In: IJCNN ’93—proc of 1993 int’l joint conf on neural networks, vol 2, pp 1339–1344

    Google Scholar 

  31. Kibler D, Aha D, Albert MK (1989) Instance-based prediction of real-valued attributes. Comput Intell 5(2):51–57

    Article  Google Scholar 

  32. Kohavi R, Wolpert D (1996) Bias plus variance decomposition for zero-one loss functions. In: Machine learning: proc of the 13th int’l conf

    Google Scholar 

  33. Koza J (1992) Genetic programming—on the programming of computers by means of natural selection. MIT Press, Cambridge

    MATH  Google Scholar 

  34. Kumar R, Rockett P (2002) Improved sampling of the Pareto-front in multiobjective genetic optimizations by steady-state evolution: a Pareto converging genetic algorithm. Evol Comput 10(3):283–314

    Article  Google Scholar 

  35. Kuncheva L, Whitaker C (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51:181–207

    Article  MATH  Google Scholar 

  36. Laumanns M, Laumanns N (2005) Evolutionary multiobjective design in automotive development. Appl Intell 23(1):55–70

    Article  Google Scholar 

  37. Lee H, Kim E, Pedrycz W (2012) A new selective neural network ensemble with negative correlation. Appl Intell 37(4):488–498

    Article  Google Scholar 

  38. Liu Y, Yao X (1999) Ensemble learning via negative correlation. Neural Netw 12(10):82–102

    Article  Google Scholar 

  39. Manderick B, Spiessens P (1989) Fine-grained parallel genetic algorithms. In: Proc of the 3rd int’l conf on genetic algorithms

    Google Scholar 

  40. Montana D (1995) Strongly typed genetic programming. Evol Comput 3(2)

  41. Niehaus J, Banzhaf W (2003) More on computational effort statistics for genetic programming. In: Genetic programming. Lecture notes in computer science, vol 2610, pp 164–172

    Chapter  Google Scholar 

  42. Opitz D, Shavlik J (1996) Generating acuurate and diverse members of a neural-network ensemble. In: Adv in Neural Information Processing Systems, vol 8. MIT Press, Cambridge

    Google Scholar 

  43. Perrone MP, Cooper LN (1993) When networks disagree: ensemble methods for hybrid neural networks. In: Neural networks for speech and image processing. Chapman-Hall, London, pp 126–142

    Google Scholar 

  44. Pierro F, Khu ST, Savic D (2007) An investigation on preference order ranking scheme for multiobjective evolutionary optimization. IEEE Trans Evol Comput 11(1):17–45

    Article  Google Scholar 

  45. Prechelt L (1994) PROBEN1—a set of neural network benchmark problems and benchmarking rules. Technical report, University of Freiburg

  46. Ruta D, Gabrys B (2002) New measure of classifier dependency in multiple classifier systems. In: Proc of the MCS’2002 conf

    Google Scholar 

  47. Ruta D, Gabrys B (2005) Classifier selection for majority voting. Inf Fusion 6(1):63–81

    Article  Google Scholar 

  48. Srinivas N, Deb K (1994) Multiobjective optimization using nondominated sorting in genetic algorithms. Evol Comput 2(3):221–248

    Article  Google Scholar 

  49. Stronger D, Stone P (2008) Polynomial regression with automated degree: A function approximator for autonomous agents. Art Intell Tools 17(1)

  50. Tanese R (1989) Distributed genetic algorithms. In: Proc of the 3rd int’l conf on genetic algorithms. Morgan Kaufmann, San Mateo, pp 434–439

    Google Scholar 

  51. Tsakonas A, Gabrys B (2012) Gradient: grammar-driven genetic programming framework for building multi-component, hierarchical predictive systems. Expert Syst Appl 39:13,253–13,266

    Article  Google Scholar 

  52. Tumer K, Ghosh J (1996) Error correlation and error reduction in ensemble classifiers. Connect Sci 8(3):385–404

    Article  Google Scholar 

  53. Yeh I-C (2008) Modeling slump of concrete with fly ash and superplasticizer. Comput Concr 5(6):559–572

    Article  Google Scholar 

  54. Zajaczkowski J, Verma B (2012) Selection and impact of different topologies in multi-layered hierarchical fuzzy systems. Appl Intell 36(3):564–584

    Article  Google Scholar 

  55. Zhang Y, Bhattacharyya S (2004) Genetic programming in classifying large-scale data: an ensemble method. Inf Sci 163(1–3):85–101

    Article  Google Scholar 

  56. Zhou A, Qu BY, Li H, Zhao SZ, Suganthan PN, Zhang Q (2011) Multiobjective evolutionary algorithms: a survey of the state of the art. Swarm Evol Comput 1:32–49

    Article  Google Scholar 

  57. Zhou Z, Wu J, Jiang Y, Chen R (2001) Genetic algorithm based selective neural network ensemble. In: Proc of 17th int’l joint conf artif intell, vol 2. Morgan Kaufmann, San Mateo, pp 797–802

    Google Scholar 

  58. Zitzler E, Künzli S (2004) Indicator-based selection in multiobjective search. In: LNCS: parallel problem solving from nature, PPSN VIII, vol 3242

    Google Scholar 

  59. Zitzler E, Laumanns M, Thiele L (1999) Spea2: improving the strength pareto evolutionary algorithm for multiobjective optimization. In: Evolutionary methods for design optimisation and control, pp 95–100

    Google Scholar 

  60. Zitzler E, Thiele L (1999) Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans Evol Comput 3(4):257–271

    Article  Google Scholar 

Download references

Acknowledgements

The author wishes to gratefully acknowledge discussions with Professor Bogdan Gabrys throughout this research. The research leading to these results has received funding from the European Commission within the Marie Curie Industry and Academia Partnerships and Pathways (IAPP) programme under grant agreement n. 251617.

Author information

Authors and Affiliations

  1. Smart Technology Research Centre, Bournemouth University, BH12 5BB, Bournemouth, UK

    Athanasios Tsakonas

Authors
  1. Athanasios Tsakonas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Athanasios Tsakonas.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tsakonas, A. An analysis of accuracy-diversity trade-off for hybrid combined system with multiobjective predictor selection. Appl Intell 40, 710–723 (2014). https://doi.org/10.1007/s10489-013-0507-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-013-0507-8

Keywords

Search

Navigation