Abstract
Extreme learning machine (ELM) randomly generates parameters of hidden nodes and then analytically determines the output weights with fast learning speed. The ill-posed problem of parameter matrix of hidden nodes directly causes unstable performance, and the automatical selection problem of the hidden nodes is critical to holding the high efficiency of ELM. Focusing on the ill-posed problem and the automatical selection problem of the hidden nodes, this paper proposes the variational Bayesian extreme learning machine (VBELM). First, the Bayesian probabilistic model is involved into ELM, where the Bayesian prior distribution can avoid the ill-posed problem of hidden node matrix. Then, the variational approximation inference is employed in the Bayesian model to compute the posterior distribution and the independent variational hyperparameters approximately, which can be used to select the hidden nodes automatically. Theoretical analysis and experimental results elucidate that VBELM has stabler performance with more compact architectures, which presents probabilistic predictions comparison with traditional point predictions, and it also provides the hyperparameter criterion for hidden node selection.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: Proceedings of international joint conference on neural networks (IJCNN2004), vol 2, (Budapest, Hungary), pp 985–990
Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501
Zhang R, Huang GB, Sundararajan N, Saratchandran P (2007) Multi-category classification using an extreme learning machine for microarray gene expression cancer diagnosis. IEEE/ACM Trans Comput Biol Bioinf 4(3):485–495
Mattos CLC, Barreto GA (2013) ARTIE and MUSCLE models: building ensemble classifiers from fuzzy ART and SOM networks. Neural Comput Appl 22(1):49–61
Huang GB, Chen L, Siew CK (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892
Han F, Huang DS (2006) Improved extreme learning machine for function approximation by encoding a priori information. Neurocomputing 69(16):2369–2373
Tang X, Han M (2009) Partial Lanczos extreme learning machine for single-output regression problems. Neurocomputing 72(13):3066–3076
Minhas R, Baradarani A, Seifzadeh S, Jonathan Wu QM (2010) Human action recognition using extreme learning machine based on visual vocabularies. Neurocomputing 73(10):1906–1917
Malathi V, Marimuthu NS, Baskar S (2010) Intelligent approaches using support vector machine and extreme learning machine for transmission line protection. Neurocomputing 73(10):2160–2167
Yeu CW, Lim MH, Huang GB, Agarwal A, Ong YS (2006) A new machine learning paradigm for terrain reconstruction. Geosci Remote Sens Lett IEEE 3(3):382–386
Huang GB, Chen L (2008) Enhanced random search based incremental extreme learning machine. Neurocomputing 71:3460–3468
Saul LK, Jaakkola TS, Jordan MI (1996) Mean field theory for Sigmoid belief networks. J Artif Intell Res 4:61–76
Yedidia JS, Freeman WT, Weiss Y (2001) Bethe free energy, Kikuchi approximations and belief propagation algorithms. Technical Report, TR-2001-16, Mitsubishi Electric Research Laboratories, Cambridge
Yedidia JS, Freeman WT, Weiss Y (2001) Generalized belief propagation. In: Advances in neural information processing systems 13, vol 14. The MIT Press, Cambridge, pp 689–695
Wainwright MJ, Jordan MI (2008) Graphical models, exponential families, and variational inference. Found Trends Mach Learn 1(1–2):1–305
Wainwright MJ, Jaakkola TS, Willsky AS (2003) Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching. In: Workshop on artificial intelligence and statistics
Beal MJ (2003) Variational algorithms for approximate Bayesian inference. Ph.D. thesis, University of Cambridge
Beal MJ, Ghahramani Z (2004) Variational Bayesian learning of directed graphical models with hidden variables. Bayesian Anal 1:1–44
Beal MJ, Ghahramani Z (2003) The variational Bayesian EM algorithm for incomplete data: with application to scoring graphical model structures. In: Bernardo JM, Dawid AP, Berger JO, West M, Heckerman D, Bayarri MJ (eds) Bayesian statistics, vol 7. Oxford University Press, Oxford, pp 453–464
Jordan MI, Ghahramani Z, Jaakkola TS, Saul LK (1999) An introduction to variational methods for graphical models. Mach Learn 37:183–233
Minka T (2005) Divergence measures and message passing. Technical report. MSR-TR-2005-173, Microsoft Research Ltd, Cambridge, UK
Bishop CM (2006) Pattern recognition and machine learning. Springer, Berlin
UIML Repository. http://archive.ics.uci.edu/ml/
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 61063035, 61402332, and the young academic team construction projects of the ‘twelve five’ integrated investment planning in Tianjin University of Science and Technology.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Chen, Y., Yang, J., Wang, C. et al. Variational Bayesian extreme learning machine. Neural Comput & Applic 27, 185–196 (2016). https://doi.org/10.1007/s00521-014-1710-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-014-1710-1