Skip to main content
SpringerLink
Log in

Optimizing connection weights in neural networks using the whale optimization algorithm

  • Foundations
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

The learning process of artificial neural networks is considered as one of the most difficult challenges in machine learning and has attracted many researchers recently. The main difficulty of training a neural network is the nonlinear nature and the unknown best set of main controlling parameters (weights and biases). The main disadvantages of the conventional training algorithms are local optima stagnation and slow convergence speed. This makes stochastic optimization algorithm reliable alternative to alleviate these drawbacks. This work proposes a new training algorithm based on the recently proposed whale optimization algorithm (WOA). It has been proved that this algorithm is able to solve a wide range of optimization problems and outperform the current algorithms. This motivated our attempts to benchmark its performance in training feedforward neural networks. For the first time in the literature, a set of 20 datasets with different levels of difficulty are chosen to test the proposed WOA-based trainer. The results are verified by comparisons with back-propagation algorithm and six evolutionary techniques. The qualitative and quantitative results prove that the proposed trainer is able to outperform the current algorithms on the majority of datasets in terms of both local optima avoidance and convergence speed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://archive.ics.uci.edu/ml/.

  2. http://www.cs.utoronto.ca/~delve/data/.

References

  • Baluja S (1994) Population-based incremental learning. A method for integrating genetic search based function optimization and competitive learning. Technical report, DTIC Document

  • Basheer IA, Hajmeer M (2000) Artificial neural networks: fundamentals, computing, design, and application. J Microbiol Methods 43(1):3–31

    Article  Google Scholar 

  • Beyer H-G, Schwefel H-P (2002) Evolution strategies-a comprehensive introduction. Natural Comput 1(1):3–52

    Article  MathSciNet  MATH  Google Scholar 

  • Blum C, Socha K (2005) Training feed-forward neural networks with ant colony optimization: an application to pattern classification. In: Hybrid intelligent systems, HIS’05, fifth international conference on IEEE, p 6

  • Braik M, Sheta A, Arieqat A (2008) A comparison between GAs and PSO in training ANN to model the TE chemical process reactor. In: AISB 2008 convention communication, interaction and social intelligence, vol 1. Citeseer, p 24

  • Chatterjee S, Sarkar S, Hore S, Dey N, Ashour AS, Balas VE (2016) Particle swarm optimization trained neural network for structural failure prediction of multistoried RC buildings. Neural Comput Appl 1–12. doi:10.1007/s00521-016-2190-2

  • Črepinšek M, Liu S-H, Mernik M (2013) Exploration and exploitation in evolutionary algorithms: a survey. ACM Comput Surv (CSUR) 45(3):35

    MATH  Google Scholar 

  • Das S, Suganthan PN (2011) Differential evolution: a survey of the state-of-the-art. IEEE Trans Evol Comput 15(1):4–31

    Article  Google Scholar 

  • Ding S, Chunyang S, Junzhao Y (2011) An optimizing BP neural network algorithm based on genetic algorithm. Artif Intell Rev 36(2):153–162

    Article  Google Scholar 

  • Dorigo M, Birattari M, Stützle T (2006) Ant colony optimization. Comput Intell Mag IEEE 1(4):28–39

    Article  Google Scholar 

  • Faris H, Aljarah I, Mirjalili S (2016) Training feedforward neural networks using multi-verse optimizer for binary classification problems. Appl Intell 45(2):322–332. doi:10.1007/s10489-016-0767-1

    Article  Google Scholar 

  • Gang X (2013) An adaptive parameter tuning of particle swarm optimization algorithm. Appl Math Comput 219(9):4560–4569

    MathSciNet  MATH  Google Scholar 

  • Goldberg DE et al (1989) Genetic algorithms in search optimization and machine learning, 412th edn. Addison-wesley, Reading Menlo Park

    MATH  Google Scholar 

  • Gupta JND, Sexton RS (1999) Comparing backpropagation with a genetic algorithm for neural network training. Omega 27(6):679–684

    Article  Google Scholar 

  • Holland JH (1992) Adaptation in natural and artificial systems. MIT Press, Cambridge

    Google Scholar 

  • Ho YC, Pepyne DL (2002) Simple explanation of the no-free-lunch theorem and its implications. J Optim Theory Appl 115(3):549–570

    Article  MathSciNet  MATH  Google Scholar 

  • Huang W, Zhao D, Sun F, Liu H, Chang E (2015) Scalable gaussian process regression using deep neural networks. In: Proceedings of the 24th international conference on artificial intelligence. AAAI Press, pp 3576–3582

  • Ilonen J, Kamarainen J-K, Lampinen J (2003) Differential evolution training algorithm for feed-forward neural networks. Neural Process Lett 17(1):93–105

    Article  Google Scholar 

  • Jianbo Y, Wang S, Xi L (2008) Evolving artificial neural networks using an improved PSO and DPSO. Neurocomputing 71(46):1054–1060

    Google Scholar 

  • Karaboga D (2005) An idea based on honey bee swarm for numerical optimization. Technical report, Technical report-tr06, Erciyes University, Engineering Faculty, Computer Engineering Department

  • Karaboga D, Gorkemli B, Ozturk C, Karaboga N (2014) A comprehensive survey: artificial bee colony (ABC) algorithm and applications. Artif Intell Rev 42(1):21–57

    Article  Google Scholar 

  • Karaboga D, Akay B, Ozturk C (2007) Artificial bee colony (ABC) optimization algorithm for training feed-forward neural networks. In: Modeling decisions for artificial intelligence. Springer, pp 318–329

  • Kennedy J (2010) Particle swarm optimization. In: Sammut C, Webb, GI (eds) Encyclopedia of machine learning. Springer, Boston, pp 760–766. doi:10.1007/978-0-387-30164-8_630

  • Kim JS, Jung S (2015) Implementation of the rbf neural chip with the back-propagation algorithm for on-line learning. Appl Soft Comput 29:233–244

    Article  Google Scholar 

  • Linggard R, Myers DJ, Nightingale C (2012) Neural networks for vision, speech and natural language, 1st edn. Springer, New York

    MATH  Google Scholar 

  • Meissner M, Schmuker M, Schneider G (2006) Optimized particle swarm optimization (OPSO) and its application to artificial neural network training. BMC Bioinform 7(1):125

    Article  Google Scholar 

  • Mendes R, Cortez P, Rocha M, Neves J (2002) Particle swarms for feedforward neural network training. In: Proceedings of the 2002 international joint conference on neural networks, IJCNN ’02, vol 2, pp 1895–1899

  • Meng X, Li J, Qian B, Zhou M, Dai X (2014) Improved population-based incremental learning algorithm for vehicle routing problems with soft time windows. In: Networking, sensing and control (ICNSC), 2014 IEEE 11th international conference on IEEE, pp 548–553

  • Mirjalili SA, Hashim SZM, Sardroudi HM (2012) Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm. Appl Math Comput 218(22):11125–11137

    MathSciNet  MATH  Google Scholar 

  • Mirjalili S (2014) Let a biogeography-based optimizer train your multi-layer perceptron. Inf Sci 269:188–209

    Article  MathSciNet  Google Scholar 

  • Mirjalili S (2015) How effective is the grey wolf optimizer in training multi-layer perceptrons. Appl Intell 43(1):150–161

    Article  Google Scholar 

  • Mirjalili S, Lewis A (2016) The whale optimization algorithm. Adv Eng Softw 95:51–67

    Article  Google Scholar 

  • Mohan BC, Baskaran R (2012) A survey: ant colony optimization based recent research and implementation on several engineering domain. Expert Syst Appl 39(4):4618–4627

    Article  Google Scholar 

  • Panchal G, Ganatra A (2011) Behaviour analysis of multilayer perceptrons with multiple hidden neurons and hidden layers. Int J Comput Theory Eng 3(2):332

    Article  Google Scholar 

  • Price K, Storn RM, Lampinen JA (2006) Differential evolution: a practical approach to global optimization. Springer, NewYork

    MATH  Google Scholar 

  • Rakitianskaia AS, Engelbrecht AP (2012) Training feedforward neural networks with dynamic particle swarm optimisation. Swarm Intell 6(3):233–270

    Article  Google Scholar 

  • Rezaeianzadeh M, Tabari H, Arabi YA, Isik S, Kalin L (2014) Flood flow forecasting using ANN, ANFIS and regression models. Neural Comput Appl 25(1):25–37

    Article  Google Scholar 

  • Sastry K, Goldberg DE, Kendall G (2014) Genetic algorithms. In: Burke EK, Kendall G (eds) Search methodologies: introductory tutorials in optimization and decision support techniques. Springer, Boston, pp 93–117. doi:10.1007/978-1-4614-6940-7_4

  • Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  • Seiffert U (2001) Multiple layer perceptron training using genetic algorithms. In: Proceedings of the European symposium on artificial neural networks, Bruges, Bélgica

  • Sexton RS, Dorsey RE, Johnson JD (1998) Toward global optimization of neural networks: a comparison of the genetic algorithm and backpropagation. Decis Support Syst 22(2):171–185

    Article  Google Scholar 

  • Sexton RS, Gupta JND (2000) Comparative evaluation of genetic algorithm and backpropagation for training neural networks. Inf Sci 129(14):45–59

    Article  MATH  Google Scholar 

  • Slowik A, Bialko M (2008) Training of artificial neural networks using differential evolution algorithm. In: Conference on human system interactions, IEEE, pp 60–65

  • Socha K, Blum C (2007) An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training. Neural Comput Appl 16(3):235–247

    Article  Google Scholar 

  • Storn R, Price K (1997) Differential evolution—a simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11(4):341–359

  • Wang L, Zeng Y, Chen T (2015) Back propagation neural network with adaptive differential evolution algorithm for time series forecasting. Expert Syst Appl 42(2):855–863

    Article  Google Scholar 

  • Wdaa ASI (2008) Differential evolution for neural networks learning enhancement. Ph.D. thesis, Universiti Teknologi, Malaysia

  • Whitley D, Starkweather T, Bogart C (1990) Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14(3):347–361

    Article  Google Scholar 

  • Wienholt W (1993) Minimizing the system error in feedforward neural networks with evolution strategy. In: ICANN93, Springer, pp 490–493

  • Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82

    Article  Google Scholar 

  • Yang X-S (ed) (2014) Random walks and optimization. In: Nature-inspired optimization algorithms, chap 3. Elsevier, Oxford, pp 45–65. doi:10.1016/B978-0-12-416743-8.00003-8

  • Zhang Y, Wang S, Ji G (2015) A comprehensive survey on particle swarm optimization algorithm and its applications. Math Probl Eng 2015:931256. doi:10.1155/2015/931256

Download references

Author information

Authors and Affiliations

  1. King Abdullah II School for Information Technology, The University of Jordan, Amman, Jordan

    Ibrahim Aljarah & Hossam Faris

  2. School of Information and Communication Technology, Griffith University, Nathan, Brisbane, QLD 4111, Australia

    Seyedali Mirjalili

Authors
  1. Ibrahim Aljarah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hossam Faris
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seyedali Mirjalili
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seyedali Mirjalili.

Ethics declarations

Conflict of interest

All authors declare that there is no conflict of interest.

Ethical standard

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Communicated by A. Di Nola.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aljarah, I., Faris, H. & Mirjalili, S. Optimizing connection weights in neural networks using the whale optimization algorithm. Soft Comput 22, 1–15 (2018). https://doi.org/10.1007/s00500-016-2442-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-016-2442-1

Keywords

Search

Navigation