Abstract
In this paper, a new and simplified hybrid algorithm mixing the simplex method of Nelder and Mead (NM) and particle swarm optimization algorithm (PSO), abbreviated as SNM-PSO, is proposed for the training of the parameters of the Artificial Neural Network (ANN). Our method differs from other hybrid PSO methods in that, \(n+1\) particles, where \(n\) is the dimension of the search space, are randomly selected (without sorting), at each iteration of the proposed algorithm for use as the initial vertices of the NM algorithm, and each such particle is replaced by the corresponding final vertex after executing the NM algorithm. All the particles are then updated using the standard PSO algorithm. Our proposed method is simpler than other similar hybrid PSO methods and places more emphasis on the exploration of the search space. Some simulation problems will be provided to compare the performances of the proposed method with PSO and other similar hybrid PSO methods in training an ANN. These simulations show that the proposed method outperforms the other compared methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahmadi MA, Ebadi M, Shokrollahi A, Majid SMJ (2013) Evolving artificial neural network and imperialist competitive algorithm for prediction oil flow rate of the reservoir. Appl Soft Comput 13(3):1085–1098
Blake C, Merz C (1998) UCI Repository of Machine Learning Databases. http://www.ics.uci.edu/~mlearn/MLRepository.html
Bertsekas D (1999) Nonlinear Programming. Athena Scientific, Belmont
Chen J, Tu X, Fu J (2005) Multilayered feed forward neural network based on particle swarm optimizer algorithm. J Syst Eng Electron 16(4):682–686
Chang CC, Lin CJ (2001) LIBSVM: A Library for Support Vector Machines. http://www.csie.ntu.edu.tw/~cjlin/libsvm
Chau KW (2007) Application of a PSO-based neural network in analysis of outcomes of construction claims. Autom Constr 16(5):642–646
Civicioglu P, Besdok E (2011) A conceptual comparison of the Cuckoo-search, particle swarm optimization, differential evolution and artificial bee colony algorithms. Artif Intell Rev 39(5):315–346
Da Y, Ge XR (2005) An improved PSO-based ANN with simulated annealing technique. Neurocomputing 63:527–533
Deng W, Li W, Yang XH (2011) A novel hybrid optimization algorithm of computational intelligence techniques for highway passenger volume prediction. Expert Syst Appl 38(5):4198–4205
Deng W, Chen R, Gao J, Song YJ, Xu JJ (2012) A novel parallel hybrid intelligence optimization algorithm for a function approximation problem. Comput Math Appl 63(1):325–336
Fan SKS, Liang YC, Zahara E (2004) Hybrid simplex search and particle swarm optimization for the global optimization for multimodal functions. Eng Optim 36(5):401–418
Goldberg DE (1989) Genetic algorithm in search optimization and machine learning. Addison-Wesley, New York
Gori M, Tesi A (1992) On the problem of local minima in back-propagation. IEEE Trans Pattern Anal 14(1):76–86
Holland J (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann Arbor
Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(7):359–366
Hsu CC, Gao CH (2008) Particle swarm optimization incorporating simplex search and center particle for global optimization. In: Proceedings of the IEEE conference on soft computing in industrial applications, pp 25–27
Juang CF (2004) A hybrid of genetic algorithm and particle swarm optimization for recurrent network design. IEEE Trans Syst Man Cybern Part B Cybern 34(3):997–1006
Juang CF, Chung IF, Hsu CH (2007) Automatic construction of feedforward/recurrent fuzzy systems by clustering-aided simplex particle swarm optimization. Fuzzy Set Syst 158(18):1979–1996
Kennedy J, Eberhart RC (1995) Particle swarm optimization. In: Proceedings of IEEE international conference on neural networks, pp 1942–1984
Liu A, Zahaya E, Yang MT (2012) A modified NM-PSO method for parameter estimation problems of models. J Appl Math, p 2012
Mendes R, Cortez P, Rocha M, Neves J (2002) Particle swarms for feedforward neural network training. In: Proceedings of the international joint conference on neural networks, pp 1895–1899
Maindonald J, John Braun W (2012) DAAG: Data Analysis And Graphics data and functions. http://CRAN.R-project.org/package=DAAG
Nelder JA, Mead R (1965) A simplex method for function minimization. Comput J 7:308–313
Nazareth L, Tseng P (2002) Gilding the lily: a variant of the Nelder–Mead algorithm based on golden-section search. Comput Optim Appl 22(1):133–144
Rusiecki A (2007) Robust LTS backpropagation learning algorithm. Lect Notes Comput Sci 4507:102–109
R Development Core Team (2011) R: A Language and Environment for Statistical Computing. http://www.R-project.org/
Shaw S, Kinsner W (1996) Chaotic simulated annealing in multilayer feedforward networks. In: Proceedings of Canadian conference on electrical and computer engineering, pp 265–269
Salerno J (1997) Using the particle swarm optimization technique to train a recurrent neural model. In: Proceedings of ninth IEEE international conference tools with artificial intelligence, pp 45–49
Trelea IC (2003) The particle swarm optimization algorithm: Convergence analysis and parameter selection. Inform Process Lett 85:317–325
Tan Y, Ruan G (2010) A three-layer back-propagation neural network for spam detection using artificial immune concentration. Soft Comput 14(3):139–150
Venkatesh YV, Kumar Raja S (2006) Multiple contour extraction from graylevel images using an artificial neural network. IEEE Trans Image Process 15(5):892–899
Yu J, Xi L, Wang S (2007) An improved particle swarm optimization for evolving feedforward artificial neural networks. Neural Process Lett 26(4):217–231
Zhang C, Shao H, Li Y (2000) Particle swarm optimization for evolving artificial neural network. In: Proceedings of the IEEE international conference on system, man, and cybernetics, pp 2487–2490
Zhang JR, Zhang J, Lok TM, Lyu MR (2007) A hybrid particle swarm optimization back-propagation algorithm for feedforward neural network training. Appl Math Comput 185(3):1026–1037
Zahaya E, Liu A (2010) Solving parameter identification problem by hybrid particle swarm optimization. In: Proceedings of the international multiconference of engineers and computer scientists, pp 36–38
Acknowledgments
This work was supported in part by the Aiming for the Top University Plan of National Chiao Tung University, the Ministry of Education, Taiwan, under Grant Number 101W9633 & 103W963, in part by the UST-UCSD International Center of Excellence in Advanced Bio-engineering sponsored by the Taiwan National Science Council I-RiCE Program under Grant Number: NSC 102-2911-I-009-101.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by T.-P. Hong.
Rights and permissions
About this article
Cite this article
Liao, SH., Hsieh, JG., Chang, JY. et al. Training neural networks via simplified hybrid algorithm mixing Nelder–Mead and particle swarm optimization methods. Soft Comput 19, 679–689 (2015). https://doi.org/10.1007/s00500-014-1292-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-014-1292-y