Skip to main content
SpringerLink
Log in

Evolving Multilayer Perceptrons

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

This paper proposes a new version of a method (G-Prop, genetic backpropagation) that attempts to solve the problem of finding appropriate initial weights and learning parameters for a single hidden layer Multilayer Perceptron (MLP) by combining an evolutionary algorithm (EA) and backpropagation (BP). The EA selects the MLP initial weights, the learning rate and changes the number of neurons in the hidden layer through the application of specific genetic operators, one of which is BP training. The EA works on the initial weights and structure of the MLP, which is then trained using QuickProp; thus G-Prop combines the advantages of the global search performed by the EA over the MLP parameter space and the local search of the BP algorithm. The application of the G-Prop algorithm to several real-world and benchmark problems shows that MLPs evolved using G-Prop are smaller and achieve a higher level of generalization than other perceptron training algorithms, such as QuickPropagation or RPROP, and other evolutive algorithms. It also shows some improvement over previous versions of the algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Fahlman, S. E.: Faster-learning variations on back-propagation: An empirical study. Proc. 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.

  2. Riedmiller, M. and Braun, H.: A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: Ruspini, H., (ed.), Proc. ICNN93, San Francisco, 1993, pp. 586–591.

  3. Merelo, J. J., Patón, M., Cañas, A., Prieto A. and Morán, F.: Optimization of a competitive learning neural network by genetic algorithms. IWANN93. Lecture Notes in Computer Science, 686 (1993), 185–192.

    Google Scholar 

  4. Castillo, P. A., González, J., Merelo, J. J., Rivas, V., Romero, G., Prieto, A.: SA-Prop: Optimization of multilayer perceptron parameters using simulated annealing. In: Lecture Notes in Computer Science, Volume I, volume 1606, 1998, pp. 661–670.

    Google Scholar 

  5. Yao, X. and Liu, Y.: Towards designing artificial neural networks by evolution. Appl. Math. Comput., 91(1) (1998), 83–90.

    Google Scholar 

  6. Liu, Y. and Yao, X.: Evolving modular neural networks which generalise well. Proc. IEEE Int. Confer. Artificial Life and Robotics. AROBIII 98, M. Sugisake, (ed.), Vol 2, 1998, pp. 736–739.

  7. Miller, G. F., Todd, P. M and Hegde, S. U.: Designing neural networks using genetic algorithms. In: J. D. Schaffer, ed., Proc. Third Int. Conf. Genetic Algorithms, San Mateo, 1989, 1989, pp 379–384.

  8. Alpaydim, E.: GAL: Networks that grow when they learn and shrink when they forget. Int. J. Pattern Recognition and Artif. Intel. 8(1) (1994), 391–414.

    Google Scholar 

  9. Balakrishnan, K. and Honavar, V.: Evolutionary design of neural architectures-a preliminary taxonomy and guide to literature. Technical report, AI Research Group, January 1995. CS-TR 95–01.

  10. Fahlman, S. and Lebière, C.: The Cascade-Correlation learning architecture. In: D. S. Touretzky, (ed.), Neural Information Processing Systems 2, Morgan-Kauffman, 1990, pp. 524–532.

  11. Parekh, R., Yang, J. and Honavar, V.: Constructive neural network learning algorithms formulti-category real-valued pattern classification. Technical Report ISU-CS-TR–97–06, Department of Computer Science, Iowa State University, 1997.

  12. Rathbun, T., Rogers, S., DeSimio, M. and Oxley, M.: MLP iterative construction algorithm. Neurocomputing 17 (1997), 195–216.

    Google Scholar 

  13. Hwang, J., You, S., Lay, S. and Jou, I.: The cascade-correlation learning: A projection pursuit learning perspective. IEEE Trans. Neural Networks 7(2) (1996), 278–289.

    Google Scholar 

  14. Jasic, T. and Poh, H.: Analysis of pruning in backpropagation networks for artificial and real world mapping problems. IWANN95. LectureNotes in Computer Science 930 (1995), 239–245.

    Google Scholar 

  15. Pelillo, M. and Fanelli, A.: A method of pruning layered feed-forward neural networks. IWANN93. Lecture Notes in Computer Science 686 (1993), 278–283.

    Google Scholar 

  16. Le Cun, Y.,Denker, J. S. and Solla, S.A.: Optimal brain damage. In: D. S. Touretzky, ed., Neural Information Systems 2, Morgan-Kauffman, 1990, pp. 598–605.

  17. Hassibi, B., Stork, D. G.,Wolff, G. andWatanabe, T.: Optimal brain surgeon: extensions and performance comparisons. In: NIPS6, 1994, pp. 263–270.

  18. Yao, X.: A review of evolutionary artificial neural networks. Technical report, CSIRO, 1992.

  19. Marín, F. J. and Sandoval, F.: Dise·o de redes neuronales artificiales mediante algoritmos genéticos. Computación Neuronal. Universidad de Santiago de Compostela, 1995, pp. 385–424.

  20. Topchy, A. P., Lebedko, O. A. and Miagkikh, V. V.: Fast learning in multilayered neural networks by means of hybrid evolutionary and gradient algorithms. To appear in Proc. of IC on Evolutionary Computation and Its Applications, Moscow, 1996.

  21. De Falco, I., Iazzetta, A., Natale, P. and Tarantino, E.: Evolutionary neural networks for nonlinear dynamics modeling. Parallel Problem Solving from Nature 98. Lecture Notes in Computer Science, 1498 (1998), 593–602.

    Google Scholar 

  22. Whitley, D., Starkweather, T. and Bogart, C.: Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallel Comp. 14 (1993), 347–361.

    Google Scholar 

  23. Schraudolph, N. N. and Belew, R. K.: Dynamic parameter encoding for genetic algorithms. Machine Learning 9(1) (1992), 9–21.

    Google Scholar 

  24. Castillo, P. A., González, J.,Merelo, J. J., Rivas, V., Romero, G. and Prieto, A.: G-Prop: global optimization of multilayer perceptrons using GAs. Submitted to Neurocomputing, 1999.

  25. Keesing, R. and Stork, D. G.: Evolution and learning in neural networks: the number and distribution of learning trials affect the rate of evolution. Adv. Neural Inform. Process. Syst. 3 (1991), 805–810.

    Google Scholar 

  26. Baldwin, J. M.: A new factor in evolution. Am. Naturalist 30 (1896), 441–451.

    Google Scholar 

  27. White, D. and Ligomenides, P.: GANNet: A genetic algorithm for optimizing topology and weights in neural network design. IWANN93. Lecture Notes in Computer Science 686 (1993), 322–327.

    Google Scholar 

  28. De Falco, I., Della Cioppa, A., Iazzetta, A., Natale, P. and Tarantino, E.: Optimizing neural networks for time series prediction. Third World Conference on Soft Computing (WSC3), June 1998.

  29. Bebis, G., Georgiopoulos, M. and Kasparis, T.: Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization. Neurocomputing 17 (1997), 167–194.

    Google Scholar 

  30. Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Complex Systems 4 (1990), 461–476.

    Google Scholar 

  31. Harp, S. A., Smad, T. and Guha, A.: Towards the genetic synthesis of neural networks. In: J. D. Schaffer, (ed.), Third Internat. Conf. Genetic Algorithms,Morgan Kauffmann, 1989, 360–369.

  32. Dodd, N., Macfarlane, D. and Marland, C.: Optimisation of artificial neural networks structure using genetic techniques implemented on multiple transputers. Proc. Transputing 91, 1991.

  33. Gruau, F. C.: Cellular enconding of genetic neural networks. Technical Report, LIP-IMAG Ecole Normale Superieure de Lyon, 46 Alleéd'Italie 69007 Lyon, France, 1992.

    Google Scholar 

  34. Petridis, V., Kazarlis, S., Papaikonomu, A. and Filelis, A.: A hybrid genetic algorithm for training neural networks. Artificial Neural Networks 2 (1992), 953–956.

    Google Scholar 

  35. Castillo, P. A., Gonzälez, J., Merelo, J. J., Rivas, V., Romero, G. and Prieto, A.: G-Prop-II: Global optimization of multilayer perceptrons using GAs. In: Congress on Evolutionary Computation, Washington D.C., 1999.

  36. Whitley, D.: The GENITOR algorithm and selection pressure: Why rank-based allocation of reproductive trials is best. In: J. D. Schaffer, (ed.), Proc. The Third Internat. Conf. Genetic Algorithms, Morgan Kauffmann Publishers, 1989, pp. 116–121.

  37. Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs, Third Extended Edition. Springer-Verlag, 1996.

  38. Kinnebrock, W.: Accelerating the standard backpropagation method using a genetic approach. Neurocomputing 6 (1994), 583–588.

    Google Scholar 

  39. Bellido, I. and Fernandez, G.: Backpropagation growing networks: Towards local minima elimination. IWANN91. Lecture Notes in Computer Science 540 (1991), 130–135.

    Google Scholar 

  40. Montana, D. J. and Davis. L.: Training feedforward neural networks using genetic algorithms. Proc. 11th Internat. Joint Conf. on Artif. Intel., 1989, pp. 762–767.

  41. Fahlman, S.: An empirical study of learning speed in back-propagation networks. Technical report, Carnegie Mellon University, 1988.

  42. Prechelt, L.: PROBEN1-A set of benchmarks and benchmarking rules for neural network training algorithms. Technical Report 21/94, Fakultät für Informatik, Universität Karlsruhe, Germany, September 1994.

    Google Scholar 

  43. Riedmiller, M.: Description and implementation details. Technical report, University of Karlsruhe, 1994.

  44. Grönroos, M. A.: Evolutionary design of neural networks. MS Thesis in Computer Science, Department of Mathematical Sciences, University of Turku, 1998.

  45. Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Technical report, Center for Machine Translation, Carnegie Mellon University, Pittsburgh and NEC Corporation, Tokyo, Pittsburgh, PA., 1990.

    Google Scholar 

  46. Mangasarian, O. L., Setiono, R. and Wolberg, W. H.: Pattern recognition via linear programming: Theory and application to medical diagnosis. In: T. F. Coleman and Y. Li, (eds), Large-scale Numerical Optimization, SIAM Publications, Philadelphia, 1990, pp. 22–30.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Architecture and Computer Technology, University of Granada, Campus de Fuentenueva, E. 18071, Granada, Spain

    P. A. Castillo, J. Carpio, J. J. Merelo, A. Prieto & G. Romero

  2. Department of Computer Science, University of Jaén, Avda. Madrid, 35, E. 23071, Jaén, Spain

    V. Rivas

Authors
  1. P. A. Castillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. Carpio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. J. Merelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. Prieto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. V. Rivas
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. G. Romero
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Castillo, P.A., Carpio, J., Merelo, J.J. et al. Evolving Multilayer Perceptrons. Neural Processing Letters 12, 115–128 (2000). https://doi.org/10.1023/A:1009684907680

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1009684907680

Search

Navigation