Skip to main content
SpringerLink
Log in

Sensitivity versus accuracy in ensemble models of Artificial Neural Networks from Multi-objective Evolutionary Algorithms

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a framework to obtain ensembles of classifiers from a Multi-objective Evolutionary Algorithm (MOEA), improving the restrictions imposed by two non-cooperative performance measures for multiclass problems: (1) the Correct Classification Rate or Accuracy (CCR) and, (2) the Minimum Sensitivity (MS) of all classes, i.e., the lowest percentage of examples correctly predicted as belonging to each class with respect to the total number of examples in the corresponding class. The proposed framework is based on collecting Pareto fronts of Artificial Neural Networks models for multiclass problems by the Memetic Pareto Evolutionary NSGA2 (MPENSGA2) algorithm, and it builds a new Pareto front (ensemble) from stored fronts. The ensemble built significantly improves the closeness to the optimum solutions and the diversity of the Pareto front. For verifying it, the performance of the new front obtained has been measured with the habitual use of weighting methodologies, such as Majority Voting, Simple Averaging and Winner Takes All. In addition to CCR and MS measures, three trade-off measures have been used to obtain the goodness of a Pareto front as a whole: Hyperarea, Laumanns’s Hyperarea (LAUMANNS) and Zitzler’s Spread (M3). The proposed framework can be adapted for any MOEA that aims to improve the compaction and diversity of its Pareto front, and whose fitness functions impose severe restrictions for multiclass problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Duda RO, Hart PE, Stork DG (2000) Pattern classification, 2nd edn. Wiley, New York

    MATH  Google Scholar 

  2. Bishop CM (2006) Pattern recognition and machine learning. Springer, New York

    MATH  Google Scholar 

  3. Kuncheva LI (2004) Combining patterns classifiers, methods and algorithms. Wiley, London

    Book  MATH  Google Scholar 

  4. Rokach L (2010) Pattern classification using Ensemble Methods. Series in machine perception artificial intelligence. World Scientific Publishing Company, Singapore

    MATH  Google Scholar 

  5. Zhou ZH (2012) Ensemble methods, foundations and algorithms. CRC Press, Boca Raton

    Book  Google Scholar 

  6. Mohapatra S, Patra D, Satpathy S (2014) An ensemble classifier system for early diagnosis of acute lymphoblastic leukemia in blood microscopic images. Neural Comput Appl 24(7–8):1887–1904

    Article  Google Scholar 

  7. Vieira DAG, Vasconcelos JA, Caminhas WM (2007) Controlling the parallel layer perceptron complexity using a multiobjective learning algorithm. Neural Comput Appl 16(4–5):317–325

    Article  Google Scholar 

  8. Ahmad F, Mat-Isa NA, Hussain Z, Sulaiman SN (2013) A genetic algorithm-based multi-objective optimization of an artificial neural network classifier for breast cancer diagnosis. Neural Comput Appl 23(5):1427–1435

    Article  Google Scholar 

  9. Deb K (2004) Multi-objective optimization using evolutionary algorithms. Wiley-interscience series in systems and optimization. John Wiley & Sons, London

    MATH  Google Scholar 

  10. Loghmanian SMR, Jamaluddin H, Ahmad R, Yusof R, Khalid M (2012) Structure optimization of neural network for dynamic system modeling using multi-objective genetic algorithm. Neural Comput Appl 21(6):1281–1295

    Article  Google Scholar 

  11. Albuquerque R, Pádua A, Takahashi RHC, Saldanha RR (2000) Improving generalization of MLPs with multi-objective optimization. Neurocomputing 35:189–194

    Article  MATH  Google Scholar 

  12. Knowles J, Watson R, Corne D (2001) Reducing local optima in single-objective problems by multi-objectivization. In: Proceedings of the 1st international conference on evolutionary multi-criterion optimization, vol 1993. Springer, pp 269–283

  13. Abbass HA (2003) Speeding up backpropagation using multiobjective evolutionary algorithms. Neural Comput 15:2705–2726

    Article  MATH  Google Scholar 

  14. Tan KC, Khor EF, Lee TH (2006) Multiobjective evolutionary algorithms and applications. Advanced information and knowledge processing. Springer, London

    Google Scholar 

  15. Ghanem AS, Venkatesh S, West G (2010) Multi-class pattern classification in imbalanced data. In: Proceedings of the 20th international conference on pattern recognition (ICPR 2010), IEEE Press, pp 2881–2884

  16. Wang S, Yao X (2012) Multiclass imbalance problems: analysis and potential solutions. IEEE Trans Syst Man Cybern B Cybern 42(4):1119–1130

    Article  Google Scholar 

  17. Provost F, Fawcett T (1997) Analysis and visualization of the classifier performance: comparison under imprecise class and cost distribution. In: Proceedings of the third international conference on knowledge discovery (KDD97) and Data Mining, California, USA, August 1997, pp 43–48

  18. Provost F, Fawcett T (1998) Robust classification system for imprecise environments. In: Proccedings of the fithteenth national conference on artificial intelligence, Chicago, USA, July 1998, pp 706–713

  19. Yao X, Islam MM (2008) Evolving artificial neural network ensembles. IEEE Comput Intell Mag 3(1):31–42

    Article  Google Scholar 

  20. Rahman MM, Islam MM, Murase K, Yao X (2016) Layered ensemble architecture for time series forecasting. IEEE Trans Cybern 46(1):270–283

    Article  Google Scholar 

  21. Wang S, Minku LL, Yao X (2015) Resampling-based ensemble methods for online class imbalance learning. IEEE Trans Knownl Data Eng 27(5):1356–1368

    Article  Google Scholar 

  22. Hsu CW, Lin CJ (2002) A comparison of methods for multi-class support vector machines. IEEE Trans Neural Netw 13(2):415–425

    Article  Google Scholar 

  23. Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45:427–437

    Article  Google Scholar 

  24. Fernández JC, Martínez FJ, Hervás C, Gutiérrez PA (2010) Sensitivity versus accuracy in multi-class problems using memetic pareto evolutionary neural networks. IEEE Trans Neural Netw 21(5):750–770

    Article  Google Scholar 

  25. Huband S, Hingston P, Barone L, While L (2006) A review of multiobjective test problems and a scalable test problem toolkit. IEEE Trans Evolut Comput 10(5):477–506

    Article  MATH  Google Scholar 

  26. Paliwal M, Kumar UA (2009) Neural networks and statistical techniques: a review of applications. Expert Syst Appl 36:2–17

    Article  Google Scholar 

  27. Rokach L (2009) Taxonomy for characterizing ensemble methods in classification tasks: a review and annotated bibliography. Comput Stat Data Anal 53:4046–4072

    Article  MATH  MathSciNet  Google Scholar 

  28. Brown G, Wyatt J, Harris R, Yao X (2005) Diversity creation methods: a survey and categorisation. Inf Fusion 6(1):5–20

    Article  Google Scholar 

  29. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51:181–207

    Article  MATH  Google Scholar 

  30. Zhenan H, Gary GY (2011) An ensemble method for performance metrics in multiobjective evolutionary algorithms . In: Proccedings of the 2011 IEEE congress on evolutionary computation (CEC), pp 1724–1729

  31. Woźniak M, Graña M, Corchado E (2014) A survey of multiple classifier systems as hybrid systems. Inf Fusion 16:3–17

    Article  Google Scholar 

  32. Theodoridis S, Koutroumbas K (2006) Pattern recognition, 3rd edn. Elsevier, Academic Press, San Diego, USA

    MATH  Google Scholar 

  33. Jin Y, Sendhoff B (2008) Pareto-based multiobjective machine learning: an overview and case studies. IEEE Trans Syst Man Cybern C Appl Rev 38(3):397–415

    Article  Google Scholar 

  34. Chandra A, Yao X (2004) DIVACE: Diverse and accurate ensemble learning algorithm. In: Proceedings of the fifth international conference on intelligent data engineering and automated learning, vol 3177 of Lectures Notes and Computer Science. Springer, Berlin, pp 619–625

  35. Abbass HA (2003) Pareto neuro-evolution: constructive ensemble of neural networks using multi-objective optimization. In: IEEE congress on evolutionary computation CEC2003, IEEE press, vol 3. Canberra, Australia, pp 2074–2080

  36. Hansen L, Salamon P (1990) Neural network emsembles. IEEE Trans Pattern Anal Mach Intell 12(10):993–1001

    Article  Google Scholar 

  37. Liu Y, Yao X (1999) Ensemble learning via negative correlation. Neural Netw 12(10):1399–1404

    Article  Google Scholar 

  38. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MATH  MathSciNet  Google Scholar 

  39. Schapire RE (1999) A brief introduction to boosting. In: Proceedings of the 16th international joint conference on artificial intelligence, vol 2, pp 1401–1406

  40. Liu Y, Yao X (1999) Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Trans Syst Man Cybern B Cybern 29(6):716–725

    Article  Google Scholar 

  41. Liu Y, Yao X (2000) Evolutionary ensembles with negative correlation learning. Trans Evolut Comput 4(4):380–387

    Article  Google Scholar 

  42. Chen H, Yao X (2009) Multi-objective neural network ensembles based on regularized negative correlation learning. IEEE Trans Knowl Data Eng 22(12):1738–1751

    Article  Google Scholar 

  43. Chen H, Yao X (2009) Regularized negative correlation learning for neural network ensembles. IEEE Trans Neural Netw 20(12):1962–1979

    Article  Google Scholar 

  44. Abbass HA (2006) Pareto-optimal approaches to neuro-ensemble learning. In: Proceeding on studies in computational intelligence (SCI), vol 16. Springer, pp 407–427

  45. Chandra A, Yao X (2006) Ensemble learning using multi-objective evolutionary algorithms. J Math Model Algorithms 5(4):417–445

    Article  MATH  MathSciNet  Google Scholar 

  46. Jin Y, Okabe T, Sendhoff B (2004) Evolutionary multi-objective approach to constructing neural network ensembles for regression, volume 1 of advances in natural computation. World Scientific, pp 653–672

  47. Garcia-Pedrajas N, Hervas-Martinez C, Ortiz-Boyer D (2005) Cooperative coevolution of artificial neural network ensembles for pattern classification. IEEE Trans Evolut Comput 9(3):271–302

    Article  Google Scholar 

  48. Minku FL, Ludemir TB (2008) Clustering and co-evolution to construct neural network ensembles: an experimental study. Neural Netw 21:1363–1379

    Article  Google Scholar 

  49. Islam MM, Yao X (2003) A constructive algorithm for training cooperative neural networks ensembles. IEEE Trans Neural Netw 14(4):820–834

    Article  Google Scholar 

  50. Jin Y, Okabe T, Sendhoff B (2004) Neural network regularization and ensembling using multi-objective evolutinary algorithms. In: Proceedings of the congress on evolutionary and ensembling using multi-objective evolutionary algorithms, vol 1. Portland, pp 1–8

  51. Fieldsend JE, Singh S (2005) Pareto evolutionary neural networks. IEEE Trans Neural Netw 16(2):338–354

    Article  Google Scholar 

  52. Jin Y, Sendholf B, Körner E (2006) Simultaneous generation of accurate and interpretable neural network classifiers. Stud Comput Intell 16:291–312

    Google Scholar 

  53. Yen GG (2006) Multiobjective evolutionary algorithm for radial basis function neural network design. Multi-objective machine learning. Studies in computational intelligence, vol 162. Springer, Berlin, Heidelberg, pp 221–239

    Chapter  Google Scholar 

  54. Dong JR, Zheng CY, Kan GY, Zhao M, Wen J, Yu J (2015) Applying the ensemble artificial neural network-based hybrid data-driven model to daily total load forecasting. Neural Comput Appl 26(3):603–611

    Article  Google Scholar 

  55. Escalante HJ, Montes y Gómez M, González JA, Gómez-Gil P, Altamirano L, Reta CA, Rosales A (2012) Acute leukemia classification by ensemble particle swarm model selection. Artif Intell Med 55:163–175

    Article  Google Scholar 

  56. Escalante HJ, Montes M, Sucar E (2010) Ensemble particle swarm model selection. In: Proceedings of the international joint conference on neural networks (IJCNN 2010), pp 1814–1824, July 2010

  57. Acosta-Mendoza N, Morales-Reyes A, Escalante HJ, Gago-Alonso A (2014) Learning to assemble classifiers via genetic programming. Int J Pattern Recognit Artif Intell 28(7):1460005

    Article  Google Scholar 

  58. Wang P, Weise T, Chiong R (2011) Novel evolutionary algorithms for supervised classification problems: an experimental study. Evolut Intell 4:3–16

    Article  Google Scholar 

  59. Wang X, Wang H (2006) Classification by evolutionary ensembles. Pattern Recognit 39(4):595–607

    Article  MATH  Google Scholar 

  60. Everson RM, Fieldsend JE (2006) Multi-class ROC analysis from a multi-objetive optimisation perspective. Pattern Recognit Lett 27:918–927

    Article  Google Scholar 

  61. Deb K, Pratab A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA2. IEEE Trans Evolut Comput 6(2):182–197

    Article  Google Scholar 

  62. Chen MR, Lu YZ, Yang G (2008) Multiobjective optimization using population-based extremal optimization. Neural Comput Appl 17(2):101–109

    Article  Google Scholar 

  63. Basseur M, Zeng RQ, Hao JK (2012) Hypervolume-based multi-objective local search. Neural Comput Appl 21(8):1917–1929

    Article  Google Scholar 

  64. Igel C, Hüsken M (2003) Empirical evaluation of the improved Rprop learning algorithms. Neurocomputing 50(6):105–123

    Article  MATH  Google Scholar 

  65. Angeline PJ, Sauders GM, Pollack JB (1994) An evolutionary algorithm that constructs recurren neural networks. IEEE Trans Neural Netw 5:54–65

    Article  Google Scholar 

  66. Lichman M (2013) UCI Machine learning repository. University of california, school of information and computer science, CA. Available online at http://archive.ics.uci.edu/ml.Irvine

  67. Fernandez-Navarro F, Hervás-Martínez C, García-Alonso C, Torres-Jiménez M (2011) Determination of relative agrarian technical efficiency by a dynamic over-sampling procedure guided by minimum sensitivity. Expert Syst Appl 38(10):12483–12490

    Article  Google Scholar 

  68. McGillivray M (2003) Commitment to development index: a critical appraisal. Technical report, AusAid

  69. Sianes A, Dorado-Moreno M, Hervás-Martínez C (2014) Rating the rich: an ordinal classification to determine which rich countries are helping poorer ones the most. Soc Indic Res 116(1):47–65

    Article  Google Scholar 

  70. Gutiérrez PA, Segovia-Vargas MJ, Salcedo-Sanz S, Hervás-Martínez C, Sanchis A, Portilla-Figueras JA, Fernandez-Navarro F (2010) Hybridizing logistic regression with product unit and RBF networks for accurate detection and prediction of banking crises. Omega 38(5):333–344

    Article  Google Scholar 

  71. Yen GG, Zhenan H (2014) Performance metrics ensemble for multiobjective evolutionary algorithms. IEEE Trans Evolut Comput 18(1):131–144

    Article  Google Scholar 

  72. Zitzler E, Thiele L (1999) Multiobjective evolutionary algorithms: a comparative case study and the strenght Pareto approach. IEEE Trans Neural Netw 3(4):414–417

    Google Scholar 

  73. Laumanns M, Zitzler E, Thiele L (2000) A unified model for multiobjective evolutionary algorithms with Elitism. In: 2000 congress on evolutionary computation, vol 1, pp 46–53

  74. Demsar J (2006) Statistical comparisons of clasiffiers over multiple data sets. J Mach Learn Res 7:1–30

    MATH  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work has been partially subsidized by the TIN2014-54583-C2-1-R project of the Spanish Ministry of Economy and Competitiveness (MINECO), FEDER funds and the P11-TIC-7508 project of the “Junta de Andalucía” (Spain).

Author information

Authors and Affiliations

  1. Department of Computer Science and Numerical Analysis, University of Cordoba, Rabanales Campus, Albert Einstein Building, 3rd Floor, 14071, Córdoba, Spain

    Juan Carlos Fernández, Manuel Cruz-Ramírez & César Hervás-Martínez

Authors
  1. Juan Carlos Fernández
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manuel Cruz-Ramírez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. César Hervás-Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Carlos Fernández.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fernández, J.C., Cruz-Ramírez, M. & Hervás-Martínez, C. Sensitivity versus accuracy in ensemble models of Artificial Neural Networks from Multi-objective Evolutionary Algorithms. Neural Comput & Applic 30, 289–305 (2018). https://doi.org/10.1007/s00521-016-2781-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-016-2781-y

Keywords

Search

Navigation