Skip to main content
SpringerLink
Log in

Hybrid back-propagation training with evolutionary strategies

  • Methodologies and Application
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

This work presents a hybrid algorithm for neural network training that combines the back-propagation (BP) method with an evolutionary algorithm. In the proposed approach, BP updates the network connection weights, and a (\(1+1\)) Evolutionary Strategy (ES) adaptively modifies the main learning parameters. The algorithm can incorporate different BP variants, such as gradient descent with adaptive learning rate (GDA), in which case the learning rate is dynamically adjusted by the stochastic (\(1+1\))-ES as well as the deterministic adaptive rules of GDA; a combined optimization strategy known as memetic search. The proposal is tested on three different domains, time series prediction, classification and biometric recognition, using several problem instances. Experimental results show that the hybrid algorithm can substantially improve upon the standard BP methods. In conclusion, the proposed approach provides a simple extension to basic BP training that improves performance and lessens the need for parameter tuning in real-world problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. The error measure \(E\) given above is just one possible measure that can be used.

  2. For the momentum coefficient \(\gamma \) this is strictly the case, while for the learning rate \(\beta \) most published works suggest values within this range.

  3. UCI Machine Learning Repository, http://archive.ics.uci.edu/ml.

  4. Normalized Root Mean Square Error.

    Table 5 Comparative result for the Mackey–Glass problem showing the average NRMSE and standard deviation
    Full size table

  5. The authors do not provide the network architecture, thus a full comparison is not possible.

References

  • Alba E, Chicano JF (2004) Training neural networks with GA hybrid algorithms. In: Deb K, et al (eds), GECCO (1). Lecture Notes in computer science, vol 3102. Springer, Berlin, pp 852–863

  • Ayinde O, Yang Y (2002) Face recognition approach based on rank correlation of Gabor-filtered images. Pattern Recognit 36(6):1275–1289

    Article  Google Scholar 

  • Cantú-Paz E, Kamath C (2005) An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Syst Man Cybern Soc 35(5):915–927

    Article  Google Scholar 

  • Castillo O, Melin P (2002) Hybrid intelligent systems for time series prediction using neural networks, fuzzy logic, and fractal theory. IEEE Trans Neural Netw 13(6):1395–1408

    Article  Google Scholar 

  • Cortez P, Cerdeira A, Almeida F, Matos T, Reis J (2009) Modeling wine preferences by data mining from physicochemical properties. Decis Support Syst 47(4):547–553

    Article  Google Scholar 

  • DeJong K (2002) Evolutionary computation: a unified approach. The MIT Press, Cambridge

  • Denoeux T (2000) A neural network classifier based on Dempster–Shafer theory. IEEE Syst Man Cybern Soc 30(2):131–150

    Article  MathSciNet  Google Scholar 

  • Dogan N, Tanrikulu Z (2010) A comparative framework for evaluating classification algorithms. In: Proceedings of the world congress on engineering 2010, vol 1. WCE, pp 379–384

  • Dorigo M, Stützle T (2004) Ant colony optimization. Bradford Company, Scituate

  • Eiben AE, Smith JE (2003) Introduction to evolutionary computing. Springer, Berlin

  • Fogel DB, Fogel LJ, Porto VW (1990) Evolving neural networks. Biol Cybern 63(6):487–493

    Article  Google Scholar 

  • Hagan M, Demuth H, Beale M (1996) Neural Network Design. PWS Publishing Company, Boston

  • Harp SA, Samad T, Guha A (1989) Towards the genetic synthesis of neural network. In: Proceedings of the third international conference on Genetic algorithms. Morgan Kaufmann Publishers Inc., San Francisco, pp 360–369

  • Hoel PG, Port SC, Stone CJ (1987) Introduction to stochastic processes. Waveland Press, Long Grove

  • Husbands P (1994) Distributed coevolutionary genetic algorithms for multi-criteria and multi-constraint optimisation. In: Selected papers from AISB workshop on evolutionary computing. Springer, London, pp 150–165

  • Isasi Viñuela P (2004) Redes neuronales artificiales: un enfoque práctico. TPearson Educacion, Upper Saddle River

  • Kim HB, Jung SH, Kim TG, Park KH (1996) Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates. Neurocomputing 11(1):101–106

    Article  MATH  Google Scholar 

  • Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2009) Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw 22(10):1448–1462

    Article  Google Scholar 

  • Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680

    Article  MATH  MathSciNet  Google Scholar 

  • Lee S-W (1996) Off-line recognition of totally unconstrained handwritten numerals using multilayer cluster neural network. IEEE Trans Pattern Anal Mach Intell 18(6):648–652

    Article  Google Scholar 

  • Mackey M, Glass L (1977) Oscillation and chaos in physiological control systems. Science 197(4300):287–289

    Article  Google Scholar 

  • Melin P, Castillo O (2005) Hybrid intelligent systems for pattern recognition using soft computing: an evolutionary approach for neural networks and fuzzy systems (Studies in fuzziness and soft computing). Springer-Verlag New York Inc., Secaucus

  • Melin P, Soto J, Castillo O, Soria J (2012) A new approach for time series prediction using ensembles of ANFIS models. Expert Syst Appl 39(3):3494–3506

    Article  Google Scholar 

  • Merelo J, Patón M, Cañas A, Prieto A, Morán F (1993) Optimization of a competitive learning neural network by genetic algorithms. In: Proceedings of the international workshop artificial neural networks (IWANN93). Lecture notes in computer science, vol 686. Morgan Kaufmann Publishers Inc., Berlin, pp 185–192

  • Miller GF, Todd PM, Hegde SU (1989) Designing neural networks using genetic algorithms. In: Proceedings of the third international conference on Genetic algorithms. Morgan Kaufmann Publishers Inc., San Francisco, pp 379–384

  • Nolfi S, Floreano D (2000) Evolutionary robotics: the biology, intelligence, and technology. MIT Press, Cambridge

  • Patel D (1996) Using genetic algorithms to construct a network for financial prediction. In: Proceedings of SPIE: applications of artificial neural networks in image processing. pp 204–213

  • Radi A, Poli R (2003) Discovering efficient learning rules for feedforward neural networks using genetic programming, chap 7. Physica-Verlag GmbH, Heidelberg, pp 133–159

  • Refaeilzadeh P, Tang L, Liu H (2009) Cross-validation. In: Liu L, Özsu MT (eds) Encyclopedia of database systems. Springer, Berlin, pp 532–538

  • Riedmiller M (1994) Rprop—description and implementation details. Technical report, University of Karlsruhe

  • Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation, chap 8. MIT Press, Cambridge, pp 318–362

  • Samanta B (2011) Prediction of chaotic time series using computational intelligence. Expert Syst Appl 38(9):11406–11411

    Article  MathSciNet  Google Scholar 

  • Samarasinghe S (2006) Neural networks for applied sciences and engineering. Auerbach Publications, Boston

  • Samaria FS, Harter AC (2002) Parameterisation of a stochastic model for human face identification. In: Applications of computer vision, vol 1994. Proceedings of the second IEEE workshop on Sarasota, FL. pp 138–142

  • Schwefel H-P (1981) Numerical optimization of computer models. John Wiley & Sons Inc., New York

    MATH  Google Scholar 

  • Stanley KO, D’Ambrosio DB, Gauci J (2009) A hypercube-based encoding for evolving large-scale neural networks. Artif Life 15(2):185–212

    Article  Google Scholar 

  • Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127

    Article  Google Scholar 

  • Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. Trans Evol Comp 1(1):67–82

    Article  Google Scholar 

  • Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447

    Article  Google Scholar 

  • Zhang G (2000) Neural networks for classification: a survey. IEEE Syst Man Cybern Soc 30(4):451–462

    Google Scholar 

  • Zhong P, Fukushima M (2007) Regularized nonsmooth newton method for multi-class support vector machines. Optim Methods Softw 22(1):225–236

    Article  MATH  MathSciNet  Google Scholar 

  • Zimmerman DW, Williams RH (1986) Modern elementary statistics, with theoretical supplement and BASIC programming. American Sciences Press, Syracuse

    MATH  Google Scholar 

Download references

Acknowledgments

First author was supported by scholarship 263888 from Consejo Nacional de Ciencia y Tecnología (CONACYT) of México. Corresponding author also thanks the Departamento de Ingeniería Eléctrica y Electrónica at the Instituto Tecnológico de Tijuana. Additionally, partial funding for this work was given by CONACYT (Mexico) Basic Science Research Grant No. 178323.

Author information

Authors and Affiliations

  1. Instituto Tecnológico de Tijuana, Av. Tecnolǵico, Fracc. Tomás Aquino, Tijuana, BC, México

    José Parra, Leonardo Trujillo & Patricia Melin

Authors
  1. José Parra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leonardo Trujillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patricia Melin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonardo Trujillo.

Additional information

Communicated by W. Pedrycz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Parra, J., Trujillo, L. & Melin, P. Hybrid back-propagation training with evolutionary strategies. Soft Comput 18, 1603–1614 (2014). https://doi.org/10.1007/s00500-013-1166-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-013-1166-8

Keywords

Search

Navigation