Skip to main content

Advertisement

SpringerLink
Log in

Designing multilayer perceptrons using a Guided Saw-tooth Evolutionary Programming Algorithm

  • Original Paper
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In this paper, a diversity generating mechanism is proposed for an Evolutionary Programming (EP) algorithm that determines the basic structure of Multilayer Perceptron classifiers and simultaneously estimates the coefficients of the models. We apply a modified version of a saw-tooth diversity enhancement mechanism recently presented for Genetic Algorithms, which uses a variable population size and periodic partial reinitializations of the population in the form of a saw-tooth function. Our improvement on this standard scheme consists of guiding saw-tooth reinitializations by considering the variance of the best individuals in the population. The population restarts are performed when the difference of variance between two consecutive generations is lower than a percentage of the previous variance. From the analysis of the results over ten benchmark datasets, it can be concluded that the computational cost of the EP algorithm with a constant population size is reduced by using the original saw-tooth scheme. Moreover, the guided saw-tooth mechanism involves a significantly lower computer time demand than the original scheme. Finally, both saw-tooth schemes do not involve an accuracy decrease and, in general, they obtain a better or similar precision.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Alcala-Fdez J, Sánchez L, García S, del Jesús MJ, Ventura S, Garrell JM, Otero J, Romero C, Bacardit J, Rivas VM, Fernández JC, Herrera F (2008) KEEL: a software tool to assess evolutionary algorithms for data mining problems. Soft Comput 13(3):307–318

    Article  Google Scholar 

  • Angeline PJ, Sauders GM, Pollack JB (1994) An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans Neural Netw 5:54–65

    Article  Google Scholar 

  • Asuncion A, Newman D (2007) UCI machine learning repository. http://www.ics.uci.edu/mlearn/MLRepository.html

  • Back T, Fogel DB, Michalewicz Z (1997) Handbook of evolutionary computation. IOP Publishing Ltd, Bristol

    Book  Google Scholar 

  • Bishop CM (2006) Pattern recognition and machine learning. Springer, Berlin

  • Bornholdt S, Graudenz D (1992) General asymmetric neural networks and structure design by genetic algorithms. Neural Netw 5(2):327–334. doi:10.1016/S0893-6080(05)80030-9

  • Bramlette MF (1991) Initialization, mutation and selection methods in genetic algorithms for function optimization. In: Belew RK, Booker LB (eds) Proceedings of the 4th international conference on genetic algorithms (ICGA’91), pp 100–107

  • Chaiyaratana N, Piroonratana T, Sangkawelert N (2007) Effects of diversity control in single-objective and multi-objective genetic algorithms. J Heuristics 13(1):1–34

    Article  Google Scholar 

  • Cobb HG, Grefenstette JJ (1993) Genetic algorithms for tracking changing environments. In: Proceedings of the 5th international conference on genetic algorithms, pp 523–530

  • De Jong ED, Watson RA, Pollack JB (2001) Reducing bloat and promoting diversity using multi-objective methods. In: Proceedings of the 2001 genetic and evolutionary computation conference, Morgan Kaufmann, San Francisco, CA, pp 11–18

  • De Jong KA (1975) An analysis of the behavior of a class of genetic adaptive systems. PhD thesis, University of Michigan, MI, USA

  • Eiben AE, Smith JE (2003) Introduction to evolutionary computing (Natural Computing Series). Springer, Berlin

  • Eshelman LJ (1991) The CHC adaptive search algorithm: How to have safe search when engaging in nontraditional genetic recombination. In: Proceedings of the 1st workshop on foundations on genetic algorithms, pp 257–266

  • Eshelman LJ, Schaffer JD (1991) Preventing premature convergence in genetic algorithms by preventing incest. In: Belew R, Booker L (eds) Proceedings of the fourth international conference on genetic algorithms, Morgan Kaufman, San Mateo, CA, pp 115–122

  • Fogel D (1995) Evolutionary computation: toward a new philosophy of machine intelligence. IEEE Press, New York

    Google Scholar 

  • Fogel D, Owens A, Wals MJ (1966) Artificial intelligence through simulated evolution. Wiley, New York

    MATH  Google Scholar 

  • Fogel DB, Fogel LJ, Porto VW (1990) Evolving neural networks. Biol Cybern 63(6):487–493. 10.1007/BF00199581

    Google Scholar 

  • Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, Reading

  • Goldberg DE, Deb K (1990) A comparative analysis of selection schemes used in genetic algorithms. In: Proceedings of the first workshop on foundations of genetic algorithms, Bloomington Campus, Indiana, USA, pp 69–93

  • Goldberg DE, Richardson J (1987) Genetic algorithms with sharing for multi-modal function optimization. In: Proceedings of the 2nd international conference on genetic algorithms, Lawrence Erlbaum Associates, pp 41–49

  • Goldberg DE, Deb K, Clark JH (1992) Genetic algorithms, noise, and the sizing of populations. Complex Syst 6(4):333–362

    MATH  Google Scholar 

  • Greewood GW, Fogel GB, Ciobanu M (1999) Emphasizing extinction in evolutionary programming. In: Angeline P, Michalewicz Z, Schoenauer M, Yao X, Zalzala A (eds) Proceedings of the 1999 congress on evolutionary computation (CEC99), pp 666–671

  • Guillén A, Rojas I, González J, Pomares H, Herrera L, Valenzuela O, Rojas F (2007) Output value-based initialization for radial basis function neural networks. Neural Process Lett 25(3):209–225

    Article  Google Scholar 

  • Gutiérrez PA, Hervás-Martínez C, Lozano M (2007) Saw-tooth algorithm guided by the variance of best individual distributions for designing evolutionary neural networks. In: Proceedings of the 8th international conference on intelligent data engineering and automated learning (IDEAL’07), Springer, Birmingham, Lecture Notes in Computer Science, vol 4881, pp 1131–1140

  • Herrera F, Lozano M (1996) Adaptation of genetic algorithm parameters based on fuzzy logic controllers. In: Herrera F, Verdegay JL (eds) Genetic algorithms and soft computing, Physica-Verlag, Heidelberg, pp 95–125

  • Herrera F, Lozano M, Verdegay JL (1996) Dynamic and heuristic fuzzy connectives-based crossover operators for controlling the diversity and convergence of real-coded genetic algorithms. Int J Intell Syst 11(12):1013–1040

    Article  MATH  Google Scholar 

  • Hervás C, Martínez-Estudillo FJ, Gutiérrez PA (2006) Classification by means of evolutionary product-unit neural networks. In: Proceedings of the 2006 international joint conference on neural networks, Vancouver, Canada, pp 2834–2842

  • Hutter M, Legg S (2006) Fitness uniform optimization. IEEE Trans Evol Comput 10(5):568–589

    Article  Google Scholar 

  • Koumousis VK, Dimou CK (2002) The effect of oscillating population size on the performance of genetic algorithms. In: Proceedings of the 4th GRACM congress on computational mechanics

  • Koumousis VK, Katsaras CP (2006) A saw-tooth genetic algorithm combining the effects of variable population size and reinitialization to enhance performance. IEEE Trans Evol Comput 10(1):19–28

    Article  Google Scholar 

  • Krishnakumar K (1989) Micro-genetic algorithms for stationary and non-stationary function optimization. In: Proceedings of the SPIE: intelligent control and adaptive systems, vol 1196, pp 289–296

  • Landwehr N, Hall M, Frank E (2005) Logistic model trees. Mach Learn 59(1–2):161–205

    Article  MATH  Google Scholar 

  • Leung FHF, Lam HK, Ling SH, Tam PKS (2003) Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Trans Neural Netw 14(1):79–88. doi:10.1109/TNN.2002.804317

    Google Scholar 

  • Levene H (1960) Essays in Honor of Harold Hotelling. In: Contributions to probability and statistics. Stanford University Press, London

  • Lozano M, Herrera F, Cano JR (2008) Replacement strategies to preserve useful diversity in steady-state genetic algorithms. Inf Sci 178(23):4421–4433

    Article  Google Scholar 

  • Mahfoud SW (1992) Crowding and preselection revisited. In: Männer R, Manderick B (eds) Parallel problem solving from nature 2, North-Holland, Amsterdam, pp 27–36

  • Maniezzo V (1993) Searching among search spaces: hastening the genetic evolution of feedforward networks. In: Proceedings of the conference on artificial neural nets and genetic algorithms

  • Martínez-Estudillo AC, Hervás-Martínez C, Martínez-Estudillo FJ, García-Pedrajas N (2006a) Hybridization of evolutionary algorithms and local search by means of a clustering method. IEEE Trans Syst Man Cybern B 36(3):534–545

    Article  Google Scholar 

  • Martínez-Estudillo AC, Martínez-Estudillo FJ, Hervás-Martínez C, García N (2006b) Evolutionary product unit based neural networks for regression. Neural Netw 19(4):477–486

    Article  MATH  Google Scholar 

  • Martínez-Estudillo FJ, Hervás-Martínez C, Gutiérrez PA, Martínez-Estudillo AC (2008) Evolutionary product-unit neural networks classifiers. Neurocomputing 72(1–2):548–561

    Article  Google Scholar 

  • McDonnell JR, Waagen DE (1993) Evolving recurrent perceptrons. In: Ruck DW (ed) Proceedings of the SPIE conference, Science of Artificial Neural Networks II, vol 1966, pp 350–361

  • Miller RG (1986) Beyond ANOVA, basics of applied statistics. Wiley, London

    Google Scholar 

  • Oppacher F, Wineberg M (1999) The shifting balance genetic algorithm: improving the GA in a dynamic environment. In: Proceedings of the genetic and evolutionary computation conference, Morgan Kaufmann, San Francisco, vol 1, pp 504–510

  • Palmes PP, Hayasaka T, Usui S (2005) Mutation-based genetic neural network. IEEE Trans Neural Netw 16(3):587–600

    Article  Google Scholar 

  • Prechelt L (1994) PROBEN1: a set of neural network benchmark problems and benchmarking rules. Tech. Rep. 21/94, Fakultät für Informatik (Universität Karlsruhe)

  • Prügel-Bennett A, Shapiro JL (1994) An analysis of genetic algorithms using statistical mechanics. Phys Rev Lett 72(9):1305–1309

    Article  Google Scholar 

  • Prügel-Bennett A, Shapiro JL (1997) The dynamics of a genetic algorithm for simple Ising systems. Physica D 104:75–114

    Article  MathSciNet  Google Scholar 

  • Rattray M (1996) Modeling the dynamics of genetic algorithms using statistical mechanics. PhD thesis, Manchester University, Manchester, UK

  • Rechenberg I (1973) Evolutionsstrategie: optimierung technischer systeme nach prinzipien der biologischen evolution. Frommann-Holzboog, Stuttgart

  • Rogers A, Pruegel-Bennett A (1999) Genetic drift in genetic algorithm selection schemes. IEEE Trans Evol Comput 3(4):298

    Article  Google Scholar 

  • Shimodaira H (2001) A diversity-control-oriented genetic algorithm DCGA: performance in function optimization. In: Proceedings of the 2001 congress on evolutionary computation, 2001, vol 1, pp 44–51. doi:10.1109/CEC.2001.934369

  • Smith RE (1993) Adaptively resizing populations: an algorithm and analysis. In: Proceedings of the fifth international conference on genetic algorithms, pp 653–653

  • Snedecor GW, Cochran WG (1989) Statistical methods, 6th edn. Iowa State University Press

  • Tsai JT, Chou JH, Liu TK (2006) Tuning the structure and parameters of a neural network by using hybrid taguchi-genetic algorithm. IEEE Trans Neural Netw 17(1):69–80

    Article  Google Scholar 

  • Tsutsui S, Fujimoto Y, Ghosh A (1997) Forking genetic algorithms: GAs with search space division schemes. Evol Comput 5(1):61–80

    Article  Google Scholar 

  • Ursem RK (2002) Diversity-guided evolutionary algorithms. In: Proceedings of parallel problem solving from nature VII (PPSN-2002), Springer, Berlin, pp 462–471

  • Ventura S, Romero C, Zafra A, Delgado JA, Hervás C (2008) JCLEC: a Java framework for evolutionary computation. Soft Comput 12(4):381–392

    Article  Google Scholar 

  • Wang L, Zheng DZ, Tang F (2002) An improved evolutionary programming for optimization. In: Proceedings of the 4th World Congress on Intelligent Control and Automation, vol 3, pp 1769–1773

  • Whitley D (1989) The GENITOR algorithm and selection pressure: why rank-based allocation of reproductive trials is best. In: Schaffer JD (ed) Proceedings of the third international conference on genetic algorithms, Morgan Kaufman, San Mateo, CA

  • Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82

    Article  Google Scholar 

  • Yao X (1997) Global optimization by evolutionary algorithms. In: Proceedings of the 2nd Aizu international symposium on parallel algorithms/architecutre synthesis, (pAs’97), Aizu-Wakamatsu, Japan, pp 282–291

  • Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447

    Article  Google Scholar 

  • Yao X, Liu Y (1997) A new evolutionary system for evolving artificial neural networks. IEEE Trans Neural Netw 8(3):694–713

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work has been partially subsidized by the TIN 2008-06681-C06-03 project of the Spanish Inter-Ministerial Commission of Science and Technology (MICYT), FEDER funds and the P08-TIC-3745 project of the “Junta de Andaluca” (Spain). The research of P. A. Gutirrez has been funded by the FPU Predoctoral Program (Spanish Ministry of Education and Science), grant reference AP2006-01746.

Author information

Authors and Affiliations

  1. Department of Computer Science and Numerical Analysis, University of Córdoba, 14071, Cordoba, Spain

    Pedro Antonio Gutiérrez & César Hervás

  2. University of Granada, 18071, Granada, Spain

    Manuel Lozano

Authors
  1. Pedro Antonio Gutiérrez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. César Hervás
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manuel Lozano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pedro Antonio Gutiérrez.

Additional information

This paper is a significant extension of a work originally reported in the Proceedings of the 8th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL’07).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gutiérrez, P.A., Hervás, C. & Lozano, M. Designing multilayer perceptrons using a Guided Saw-tooth Evolutionary Programming Algorithm. Soft Comput 14, 599–613 (2010). https://doi.org/10.1007/s00500-009-0429-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-009-0429-x

Keywords

Search

Navigation