Abstract
As a novel learning algorithm for single-hidden-layer feedforward neural networks, extreme learning machines (ELMs) have been a promising tool for regression and classification applications. However, it is not trivial for ELMs to find the proper number of hidden neurons due to the nonoptimal input weights and hidden biases. In this paper, a new model selection method of ELM based on multi-objective optimization is proposed to obtain compact networks with good generalization ability. First, a new leave-one-out (LOO) error bound of ELM is derived, and it can be calculated with negligible computational cost once the ELM training is finished. Furthermore, the hidden nodes are added to the network one-by-one, and at each step, a multi-objective optimization algorithm is used to select optimal input weights by minimizing this LOO bound and the norm of output weight simultaneously in order to avoid over-fitting. Experiments on five UCI regression data sets are conducted, demonstrating that the proposed algorithm can generally obtain better generalization performance with more compact network than the conventional gradient-based back-propagation method, original ELM and evolutionary ELM.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Huang GB, Zhu QY, Siew C (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: Proceedings of International Joint Conference on Neural Networks (IJCNN2004), vol 2, pp 25–29
Huang GB, Zhu QY, Siew C (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501
Chen FL, Ou TY (2011) Sales forecasting system based on gray extreme learning machine with taguchi method in retail industry. Expert Syst Appl 38:1336–1345
Minhas R, Baradarani A, Seifzadeh S, Wu QMJ (2010) Human action recognition using extreme learning machine based on visual vocabularies. Neurocomputing 73:1906–1917
Mohammed A, Wu QMJ, Sid-Ahmed M (2010) Application of wave atoms decomposition and extreme learning machine for fingerprint classification. Lect Notes Comput Sci 6112:246–256
Huang GB, Wang D, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2:107–122
Feng GR, Huang GB, Lin QP, Gay R (2009) Error minimized extreme learning machine with growth of hidden nodes and incremental learning. IEEE Trans Neural Netw 20:1352–1357
Lan Y, Soh Y, Huang GB (2010) Random search enhancement of error minimized extreme learning machine. In: Proceedings of European Symposium on Artificial Neural Networks (ESANN 2010), pp 327–332
Li K, Huang GB, Ge SS (2010) Fast construction of single hidden layer feedforward networks. In: Rozenberg G, Bäck T, Kok JN (eds) Handbook of Natural Computing. Springer, Berlin
Lan Y, Soh Y, Huang GB (2006) Two-stage extreme learning machine for regression. Neurocomputing 73:3028–3038
Zhu QY, Qin AK, Suganthan P, Huang GB (2005) Evolutionary extreme learning machine. Pattern Recognit 38:1759–1763
Huang GB, Ding XJ, Zhou H (2010) Optimization method based extreme learning machine for classification. Neurocomputing 74:155–163
Cawley G, Talbot N (2007) Preventing over-fitting in model selection via bayesian regularisation of the hyper-parameters. J Mach Learn Res 8:841–861
Huang GB, Babri H (1998) Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans Neural Netw 9:224–229
Girosi F, Jones M, Poggio T (1995) Regularization theory and neural network architectures. Neural Comput 7:219–269
Evgeniou T, Pontil M, Poggio T (2000) Regularization networks and support vector machines. Adv Comput Math 13:1–50
Frénay B, Verleysen M (2010) Using SVMs with randomised feature spaces: an extreme learning approach. In: Proceedings of The 18th European Symposium on Artificial Neural Networks (ESANN 2010), pp 315–320
Myers R (1990) Classical and modern regression with applications. Duxbury Press, CA
Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence (IJCAI), pp 1137–1143
Bartlett P (1998) The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Trans Inf Theory 44:525–536
Huang VL, Suganthan P, Liang JJ (2006) Comprehensive learning particle swarm optimizer for solving multiobjective optimization problems. Int J Intell Syst 21:209–226
Zhu QY, Huang GB (2004) Source codes of ELM algorithm. In: http://www.ntu.edu.sg/home/egbhuang/, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
Newman D, Hettich S, Blake C (1998) UCI Repository of machine learning databases. In: http://www.ics.uci.edu/mlearn/MLRepository.html, Department of Information and Computer Science, University of California, Irvine, CA
Wang G, Yeung D, Lochovsky F (2008) A new solution path algorithm in support vector regression. IEEE Trans Neural Netw 19:1753–1767
Acknowledgments
This work was supported by the National Natural Science Foundation of China (60873104). We also thank the author Suganthan of [21] for providing implementation of MOCLPSO.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Mao, W., Tian, M., Cao, X. et al. Model selection of extreme learning machine based on multi-objective optimization. Neural Comput & Applic 22, 521–529 (2013). https://doi.org/10.1007/s00521-011-0804-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-011-0804-2