Abstract
Extreme learning machine (ELM) is a novel and recent machine learning algorithm which was first proposed by Huang et al. (Proceedings of 2004 IEEE international joint conference on, pp 985–990, 2004). Over the last decade, ELM has gained a remarkable research interest with tremendous audiences from different domains in a short period of time due to its impressive characteristics over other single hidden-layer feedforward neural networks. Although ELM enjoys powerful advantages, it still has some potential weaknesses like performance sensitivity to the initial condition of the input weights, number of hidden neurons, and the selection of activation functions. In order to overcome the limitations of classical ELM, many metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, memetic and trajectory algorithms have been proposed for optimizing the different components of ELM by researchers aiming to improve the generalization performance of ELM networks for different types of complex problems and applications. Therefore our review paper intent to conduct a deep study of the important aspects of applying metaheuristic algorithms for optimizing ELM networks. Three main streams of research lines are identified: the optimization of input weights and hidden biases, selection of hidden neurons, and optimization of activation functions. Furthermore, this paper will discuss a wide spectrum of applications of metaheuristic-based ELM models. We will highlight the strengths of these models and the improvements that are suggested in the literature to overcome their weaknesses. We touch upon several interesting and challenging open issues in optimizing ELM using metaheuristics.









Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alade OA , Selamat A, Sallehuddin R (2017) A review of advances in extreme learning machine techniques and its applications. In: International conference of reliable information and communication technology. Springer, pp 885–895
Alexandridis A, Famelis IT, Tsitouras C, Simos T, Tsitouras C (2016) Particle swarm optimization for complex nonlinear optimization problems. In: AIP conference proceedings, vol 1738. AIP Publishing, pp 480120
Aljarah I, Ludwig SA (2012) Parallel particle swarm optimization clustering algorithm based on mapreduce methodology. In: Nature and biologically inspired computing (NaBIC), 2012 fourth world congress on. IEEE, pp 104–111
Aljarah I, Ludwig SA (2013) A new clustering approach based on glowworm swarm optimization. In: Evolutionary computation (CEC), 2013 IEEE congress on. IEEE, pp 2642–2649
Almeida LB (1989) Backpropagation in perceptrons with feedback. In: Neural computers. Springer, pp199–208
Arunadevi B, Deepa SN (2013) Brain tumor tissue categorization in 3D magnetic resonance images using improved PSO for extreme learning machine. Progress Electromagn Res B 49:31–54
Azzini A, Tettamanzi AGB (2011) Evolutionary anns: a state of the art survey. Intell Artif 5(1):19–35
Bartlett PL (1998) The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Trans Inf Theory 44(2):525–536
Bi X, Zhao X, Wang G, Zhang P, Wang C (2015) Distributed extreme learning machine with kernels based on mapreduce. Neurocomputing 149:456–463
Leonora B, Marco D, Gambardella LM, Gutjahr WJ (2009) A survey on metaheuristics for stochastic combinatorial optimization. Nat Comput 8(2):239–287
Blum C, Roli A (2003) Metaheuristics in combinatorial optimization: overview and conceptual comparison. ACM Computi Surv 35(3):268–308
BoussaïD I, Lepagnot J, Siarry P (2013) A survey on optimization metaheuristics. Inf Sci 237:82–117
Cao J, Lin Z (2015) Extreme learning machines on high dimensional and large data applications: a survey. Math Probl Eng
Cao J, Lin Z, Huang G-B (2012) Self-adaptive evolutionary extreme learning machine. Neural Process Lett 36(3):285–305
Cho J-H, Lee D-J, Chun M-G (2007) Parameter optimization of extreme learning machine using bacterial foraging algorithm. J Korean Ins Intell Syst 17(6):807–812
Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Math Control Signals Syst 2(4):303–314
de Oliveira JFL, Ludermir TB (2012) An evolutionary extreme learning machine based on fuzzy fish swarms. In: Proceedings on the international conference on artificial intelligence (ICAI), p 1. The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp)
Ding S, Chunyang S, Junzhao Y (2011) An optimizing bp neural network algorithm based on genetic algorithm. Artif Intell Rev 36(2):153–162
Ding S, Xinzheng X, Nie R (2014) Extreme learning machine and its applications. Neural Comput Appl 25(3–4):549–556
Ding S, Zhao H, Zhang Y, Xinzheng X, Nie R (2015) Extreme learning machine: algorithm, theory and applications. Artif Intell Rev 44(1):103–115
Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern Part B Cybern 26(1):29–41
Eberhart R, Kennedy J (1995) A new optimizer using particle swarm theory. In: Micro machine and human science, 1995. MHS’95., Proceedings of the sixth international symposium on. IEEE, pp 39–43
Ertuğrul ÖF, Kaya Y (2014) A detailed analysis on extreme learning machine and novel approaches based on elm. Am J Comput Sci Eng 1(5):43–50
Eshtay M, Faris H, Obeid N (2018) Improving extreme learning machine by competitive swarm optimization and its application for medical diagnosis problems. Expert Syst Appl 104:134
Faris H, Aljarah I, Mirjalili S (2016) Training feedforward neural networks using multi-verse optimizer for binary classification problems. App Intell 45(2):322–332
Faris H, Aljarah I, Al-Madi N, Mirjalili S (2016) Optimizing the Learning Process of Feedforward Neural Networks Using Lightning Search Algorithm. Int J Artif Intell Tools 25(06):1650033
Feo TA, Resende MGC (1995) Greedy randomized adaptive search procedures. J Glob Optim 6(2):109–133
Freire A, Barreto G (2014) A new model selection approach for the elm network using metaheuristic optimization. In: European symposium on artificial neural networks, computational intelligence and machine learning (ESANN)
Glover F (1989) Tabu search part I. ORSA J Comput 1(3):190–206
Gori M, Tesi A (1992) On the problem of local minima in backpropagation. IEEE Trans Pattern Anal Mach Intell 1:76–86
Gupta JND, Sexton RS (1999) Comparing backpropagation with a genetic algorithm for neural network training. Omega 27(6):679–684
Hagan MT, Menhaj MB (1994) Training feedforward networks with the Marquardt algorithm. IEEE Trans Neural Netw 5(6):989–993
Han F, Yao H-F, Ling Q-H (2013) An improved evolutionary extreme learning machine based on particle swarm optimization. Neurocomputing 116:87–93
Hart WE, Krasnogor N, Smith JE (2004) Recent advances in memetic algorithms, volume 166. Springer, Berlin
Ho Y-C, Pepyne DL (2002) Simple explanation of the no-free-lunch theorem and its implications. J Optim Theory Appl 115(3):549–570
Hoerl AE, Kennard RW (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1):55–67
Holland JH (1992) Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT Press, Cambridge
Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc Nat Acad Sci 79(8):2554–2558
Hornik K (1991) Approximation capabilities of multilayer feedforward networks. Neural Netw 4(2):251–257
Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366
Huang G, Huang G-B, Song S, You K (2015) Trends in extreme learning machines: a review. Neural Netw 61:32–48
Huang G-B (2015) What are extreme learning machines? Filling the gap between frank Rosenblatts dream and John von Neumanns puzzle. Cogn Comput 7(3):263–278
Huang G-B, Babri HA (1998) Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans Neural Netw 9(1):224–229
Huang G-B, Bai Z, Kasun LLC, Vong CM (2015) Local receptive fields based extreme learning machine. IEEE Comput Intell Mag 10(2):18–29
Huang G-B, Chen L, Siew CK et al (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892
Guang-Bin H, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122
Huang G-B, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B Cybern 42(2):513–529
Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: Neural networks, 2004. Proceedings of 2004 IEEE international joint conference on, vol 2. IEEE, pp 985–990
Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501
Huang Q, Jiang C, Huang Y (2016) The prediction method of SO\(\_2\) concentration in sulfuric acid production process based on GA-ELM. In: Intelligent human-machine systems and cybernetics (IHMSC), 2016 8th international conference on, vol 2. IEEE, pp 140–143
Huang Y, Lai D (2012) Hidden node optimization for extreme learning machine. Aasri Procedia 3:375–380
Huynh HT, Won Y (2008) Evolutionary algorithm for training compact single hidden layer feedforward neural networks. In: International joint conference on neural networks (IJCNN). IEEE, pp 3028–3033
Karaboga D (2005) An idea based on honey bee swarm for numerical optimization. Technical report, Technical report-tr06, Erciyes University, Engineering Faculty, Computer Engineering Department
Karlik B, Olgac AV (2011) Performance analysis of various activation functions in generalized MLP architectures of neural networks. Int J Artif Intell Expert Syst 1(4):111–122
Kasun LLC, Zhou H, Huang G-B, Vong CM (2013) Representational learning with elms for big data
Kaya Y, Kayci L, Tekin R, Ertuğrul ÖF (2014) Evaluation of texture features for automatic detecting butterfly species using extreme learning machine. J Exp Theor Artif Intell 26(2):267–281
Kirkpatrick S, Gelatt CD, Vecchi MP et al (1983) Optimization by simulated annealing. Science 220(4598):671–680
Kohonen T (1990) The self-organizing map. Proc IEEE 78(9):1464–1480
Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection, vol 1. MIT Press, Cambridge
Krasnogor N, Smith J (2005) A tutorial for competent memetic algorithms: model, taxonomy, and design issues. IEEE Trans Evol Comput 9(5):474–488
Leshno M, Lin VY, Pinkus A, Schocken S (1993) Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw 6(6):861–867
Li B, Li Y, Rong X (2013) The extreme learning machine learning algorithm with tunable activation function. Neural Comput Appl 22(3–4):531–539
Li S, Wang P, Goel L (2015) Short-term load forecasting by wavelet transform and evolutionary extreme learning machine. Electr Power Syst Res 122:96–103
Li X, Shao Z, Qian J et al (2002) An optimizing method based on autonomous animats: fish-swarm algorithm. Syst Eng Theory Pract 22(11):32–38
Lin M, Zhang C, Su C (2016) Prediction of effluent from WWTPS using differential evolutionary extreme learning machines. In: Control conference (CCC), 2016 35th Chinese. IEEE, pp 2034–2038
Liu N, Wang H (2013) Evolutionary extreme learning machine and its application to image analysis. J Signal Proces Syst 73(1):73–81
Malhotra R, Singh N, Singh Y (2011) Genetic algorithms: concepts, design for optimization of process controllers. Comput Inf Sci 4(2):39
Matias T, Araújo R, Antunes CH, Gabriel D (2013) Genetically optimized extreme learning machine. In: Emerging technologies and factory automation (ETFA), 2013 IEEE 18th conference on. IEEE, pp 1–8
Matias T, Souza F, Arajo R, Antunes CH (2014) Learning of a single-hidden layer feedforward neural network using an optimized extreme learning machine. Neurocomputing 129:428–436
Mladenović N, Hansen P (1997) Variable neighborhood search. Comput Oper Res 24(11):1097–1100
Mohapatra P, Chakravarty S, Dash PK (2015) An improved cuckoo search based extreme learning machine for medical data classification. Swarm Evol Comput 24:25–49
Moscato P (1989) On evolution, search, optimization, genetic algorithms and martial arts: towards memetic algorithms. Technical Report C3P Report 826, California Institute of Technology
Nahvi B, Habibi J, Mohammadi K, Shamshirband S, Razgan OSA (2016) Using self-adaptive evolutionary algorithm to improve the performance of an extreme learning machine for estimating soil temperature. Comput Electron Agric 124:150–160
Niu P, Ma Y, Li M, Yan S, Li G (2016) A kind of parameters self-adjusting extreme learning machine. Neural Process Lett 44(3):813–830
Ojha VK, Abraham A, Snášel V (2017) Metaheuristic design of feedforward neural networks: a review of two decades of research. Eng Appl Artif Intell 60:97–116
Salam MA, Hegazy O, Soliman OS (2015) Fpa-elm model for stock market prediction. Int J Adv Res Comput Sci Softw Eng 5(2):1050–1063
Pacifico LDS, Ludermir TB (2006) Evolutionary extreme learning machine based on particle swarm optimization and clustering strategies. In: International joint conference on neural networks (IJCNN). IEEE, pp 1–6
Pacifico LDS, Ludermir TB (2013) Evolutionary extreme learning machine based on particle swarm optimization and clustering strategies. In: Neural networks (IJCNN), the 2013 international joint conference on. IEEE, pp 1–6
Passino KM (2002) Biomimicry of bacterial foraging for distributed optimization and control. IEEE Control Syst 22(3):52–67
Qin-Yu Z, Qin AK, Suganthan PN, Huang G-B (2005) Evolutionary extreme learning machine. Pattern recognition 38(10):1759–1763
Rosenblatt F (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev 65(6):386
Rumelhart DE, Hinton GE, Williams RJ (1988) Neurocomputing: foundations of research chapter Learning Representations by back-propagating errors. MIT Press, Cambridge, pp 696–699
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning international representations by error propagation. In: Rumelhart DE, McCleland JL (eds) Parallel distributed processing: exploration in the microstructure of cognition, vol 1, Chap 8. MIT Press, Cambridge
Salam MA, Zawbaa HM, Emary E, Ghany KKA, Parv B (2016) A hybrid dragonfly algorithm with extreme learning machine for prediction. In: INnovations in Intelligent SysTems and Applications (INISTA), 2016 international symposium on. IEEE, pp 1–6
Sánchez-Monedero J, Hervas-Martinez C, Gutiérrez PA, Ruz MC, Moreno MCR, Cruz-Ramirez M (2010) Evaluating the performance of evolutionary extreme learning machines by a combination of sensitivity and accuracy measures. Neural Netw World 20(7):899
Sattar AMA, Erturul ÖF, Gharabaghi B, McBean EA, Cao J (2017) Extreme learning machine model for water network management. J Neural Comput Appl 2017:1–13
Schuh MA, Angryk RA, Sheppard JW (2012) Evolving kernel functions with particle swarms and genetic programming. In: FLAIRS conference
Schwefel H-P (1987) Collective phenomena in evolutionary systems
Silva DNG, Pacifico LDS, Ludermir TB (2011) An evolutionary extreme learning machine based on group search optimization. In: Congress on evolutionary computation (CEC). IEEE, pp 574–580
Storn R, Price K (1997) Differential evolutiona simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11(4):341–359
Stützle TG (1999) Local search algorithms for combinatorial problems: analysis, improvements, and new applications, vol 220. Infix Sankt Augustin
Sun X, Qin L (2014) An extreme learning machine based on quantum particle swarm optimization and its application in handwritten numeral recognition. In: Software engineering and service science (ICSESS), 2014 5th IEEE international conference on. IEEE, pp 323–326
Sun Z-L, Choi T-M, Kin-Fan A, Yong Y (2008) Sales forecasting using extreme learning machine with applications in fashion retailing. Decis Support Syst 46(1):411–419
Suresh S, Babu RV, Kim HJ (2009) No-reference image quality assessment using modified extreme learning machine classifier. Appl Soft Comput 9(2):541–552
Tang J, Deng C, Huang G-B (2016) Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst 27(4):809–821
Voudouris C (1997) Guided local search for combinatorial optimisation problems. PhD Thesis, University of Essex
Wang J, Ye K, Cao J, Wang T, Xue A, Cheng Y, Yin C (2017) Doa estimation of excavation devices with elm and music-based hybrid algorithm. Cogn Comput 9:1–17
Wang Y, Cao F, Yuan Y (2011) A study on effectiveness of extreme learning machine. Neurocomputing 74(16):2483–2490
Werbos PJ (1974) Beyond regression: new tools for prediction and analysis in the behavioral science. Ph. D. Thesis, Harvard University
Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82
Xu X, Ji Z, Yuan F, Liu X (2014) A novel parallel approach of cuckoo search using mapreduce. In: 2014 international conference on computer, communications and information technology (CCIT 2014). Atlantis Press
Xue B, Ma X, Gu J, Li Y (2013) An improved extreme learning machine based on variable-length particle swarm optimization. In: International conference on robotics and biomimetics (ROBIO). IEEE, pp 1030–1035
Yang X-S (2010) Nature-inspired metaheuristic algorithms. Luniver Press, Bristol
You X, Shu Y (2006) Evolutionary extreme learning machine-based on particle swarm optimization. Adv Neural Netw ISNN 2006:644–652
Yang H, Yi J, Zhao J, Dong ZY (2013) Extreme learning machine based genetic algorithm and its application in power system economic dispatch. Neurocomputing 102:154–162
Yang Z, Wen X, Wang Z (2015) Qpso-elm: an evolutionary extreme learning machine based on quantum-behaved particle swarm optimization. In: International conference on advanced computational intelligence (ICACI). IEEE, pp 69–72
Yang Z, Zhang T, Zhang D (2016) A novel algorithm with differential evolution and coral reef optimization for extreme learning machine training. Cogn Neurodyn 10(1):73–83
Zhai J, Hong-yu X, Wang X (2012) Dynamic ensemble extreme learning machine based on sample entropy. Soft Comput 16(9):1493–1502
Zhai J, Zang L, Zhou Z (2018) Ensemble dropout extreme learning machine via fuzzy integral for data classification. Neurocomputing 275:1043–1052
Zhai J, Zhang S, Wang C (2017) The classification of imbalanced large data sets based on mapreduce and ensemble of elm classifiers. Int J Mach Learn Cybern 8(3):1009–1017
Zhang Y, Cai Z, Wu J, Wang X, Liu X (2015) A memetic algorithm based extreme learning machine for classification. In: Neural networks (IJCNN), 2015 international joint conference on neural networks. IEEE, pp 1–8
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interest regarding the publication of this paper.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Eshtay, M., Faris, H. & Obeid, N. Metaheuristic-based extreme learning machines: a review of design formulations and applications. Int. J. Mach. Learn. & Cyber. 10, 1543–1561 (2019). https://doi.org/10.1007/s13042-018-0833-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-018-0833-6