Abstract
Extreme learning machine (ELM) is a kind of random projection-based neural networks, whose advantages are fast training speed and high generalization. However, three issues can be improved in ELM: (1) the calculation of output weights takes \(O\left( {L^{2}N} \right) \) time (with N training samples and L hidden nodes), which is relatively slow to train a model for large N and L; (2) the manual tuning of L is tedious, exhaustive and time-consuming; (3) the redundant or irrelevant information in the hidden layer may cause overfitting and may hinder high generalization. Inspired from compressive sensing theory, we propose an efficient ELM via very sparse random projection (VSRP) called VSRP-ELM for training with large N and L. The proposed VSRP-ELM adds a novel compression layer between the hidden layer and output layer, which compresses the dimension of the hidden layer from \(N\times L\) to \(N\times k \,(\hbox {where } k<L)\) under projection with random sparse-Bernoulli matrix. The advantages of VSRP-ELM are (1) faster training time \(O\left( {k^{2}N} \right) , k<L,\) is obtained for large L; (2) the tuning time of L can be significantly reduced by initializing a large L, and then shrunk to k using just a few trials, while maintaining a comparable result of the original model accuracy; (3) higher generalization may be benefited from the cleaning of redundant or irrelevant information through VSRP. From the experimental results, the proposed VSRP-ELM can speed ELM up to 7 times, while the accuracy can be improved up to 6%.



Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Achlioptas D (2003) Database-friendly random projections: Johnson–Lindenstrauss with binary coins. J Comput Syst Sci 66:671–687. https://doi.org/10.1016/S0022-0000(03)00025-4
Bartlett PL (1998) The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Trans Inf Theory 44:525–536
Calderbank R, Jafarpour S, Schapire R (2009) Compressed learning: universal sparse dimensionality reduction and learning in the measurement domain. Technical report, Princeton University. https://pdfs.semanticscholar.org/627c/14fe9097d459b8fd47e8a901694198be9d5d.pdf. Accessed 14 Mar 2017
Candes EJ, Tao T (2005) Decoding by linear programming. IEEE Trans Inf Theory 51:4203–4215. https://doi.org/10.1109/Tit.2005.858979
Candes EJ, Tao T (2006) Near-optimal signal recovery from random projections: universal encoding strategies. IEEE Trans Inf Theory 52:5406–5425. https://doi.org/10.1109/Tit.2006.885507
Choi K, Toh KA, Byun H (2011) Realtime training on mobile devices for face recognition applications. Pattern Recognit 44:386–400
Choi K, Toh KA, Uh Y, Byun H (2012) Service-oriented architecture based on biometric using random features and incremental neural networks. Soft Comput 16:1539–1553
Ding S, Zhang N, Zhang J, Xu X, Shi Z (2017) Unsupervised extreme learning machine with representational features. Int J Mach Learn Cybern 8:587–595
He Q, Jin X, Du C, Zhuang F, Shi Z (2014) Clustering in extreme learning machine feature space. Neurocomputing 128:88–95
Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B Cybern 42:513–529
Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501. https://doi.org/10.1016/j.neucom.2005.12.126
Kabán A (2014) New bounds on compressive linear least squares regression. In: AISTATS, pp 448–456
Kasun LLC, Zhou H, Huang GB, Vong CM (2013) Representational learning with ELMs for big data. IEEE Intell Syst 28:31–34
Kim Y, Toh KA (2008) Sparse random projection for efficient cancelable face feature extraction. In: Proceedings of the IEEE conference on industrial electronics and applications, pp 2139–2144
Li P, Hastie TJ, Church KW (2006) Very sparse random projections. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 287–296
Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml. Accessed 16 June 2016
Liu L, Fieguth P (2012) Texture classification from random features. IEEE Trans Pattern Anal Mach Intell 34:574–586
Liu M, Liu B, Zhang C, Wang W, Sun W (2017) Semi-supervised low rank kernel learning algorithm via extreme learning machine. Int J Int J Mach Learn Cyber 8:1039–1052
Lu Y, Dhillon P, Foster DP, Ungar L (2013) Faster ridge regression via the subsampled randomized hadamard transform. In: Advances in neural information processing systems, pp 369–377
Luo J, Vong CM, Wong PK (2014) Sparse Bayesian extreme learning machine for multi-classification. IEEE Trans Neural Netw Learn Syst 25:836–843
Mao W, Wang J, Xue Z (2017) An ELM-based model with sparse-weighting strategy for sequential data imbalance problem. Int J Mach Learn Cybern 8:1333–1345
Miche Y, Sorjamaa A, Bas P, Simula O, Jutten C, Lendasse A (2010) OP-ELM: optimally pruned extreme learning machine. IEEE Trans Neural Netw Learn Syst 21:158–162
Minhas R, Baradarani A, Seifzadeh S, Wu QJ (2010) Human action recognition using extreme learning machine based on visual vocabularies. Neurocomputing 73:1906–1917
Mohammed AA, Minhas R, Wu QJ, Sid-Ahmed MA (2011) Human face recognition based on multidimensional PCA and extreme learning machine. Pattern Recognit 44:2588–2597
Pan C, Park DS, Yang Y, Yoo HM (2012) Leukocyte image segmentation by visual attention and extreme learning machine. Neural Comput Appl 21:1217–1227
Paul S, Boutsidis C, Magdon-Ismail M, Drineas P (2013) Random projections for support vector machines. In: Artificial intelligence and statistics, pp 498–506
Rong H-J, Ong Y-S, Tan A-H, Zhu Z (2008) A fast pruned-extreme learning machine for classification problem. Neurocomputing 72:359–366
Rong H-J, Suresh S, Zhao G-S (2011) Stable indirect adaptive neural controller for a class of nonlinear system. Neurocomputing 74:2582–2590
Rong H-J, Zhao G-S (2013) Direct adaptive neural control of nonlinear systems with extreme learning machine. Neural Comput Appl 22:577–586
Tang J, Deng C, Huang GB (2016) Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst 27:809–821
Thanei GA, Heinze C, Meinshausen N (2017) Random projections for large-scale regression. In: Big and complex data analysis, pp 51–68
Vanschoren J, Van Rijn JN, Bischl B, Torgo L (2014) OpenML: networked science in machine learning. ACM SIGKDD Explor Newslett 15:49–60
Vempala SS (2004) The random projection method. American Mathematical Society, Providence
Wan S, Mak MW, Kung SY (2014a) R3P-Loc: a compact multi-label predictor using ridge regression and random projection for protein subcellular localization. J Theor Biol 360:34–45
Wan S, Mak MW, Zhang B, Wang Y, Kung S-Y (2014b) Ensemble random projection for multi-label classification with application to protein subcellular localization. In: IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 5999–6003
Wang R, Wang X-Z, Kwong S, Xu C (2017a) Incorporating diversity and informativeness in multiple-instance active learning. IEEE Trans Fuzzy Syst 25:1460–1475
Wang X-Z, Wang R, Xu C (2017) Discovering the relationship between generalization and uncertainty by incorporating complexity of classification. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2017.2653223
Williams D, Hinton G (1986) Learning representations by back-propagating errors. Nature 323:533–538
Wong CM, Vong CM, Wong PK, Cao J (2016) Kernel-based multilayer extreme learning machines for representation learning. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2016.2636834
Yan Y-T, Zhang Y-P, Zhang Y-W, Du X-Q (2017) A selective neural network ensemble classification for incomplete data. Int J Mach Learn Cybern 8:1513–1524
Zhai J, Zhang S, Wang C (2017) The classification of imbalanced large data sets based on mapreduce and ensemble of elm classifiers. Int J Mach Learn Cybern 8:1009–1017
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Communicated by X. Wang, A. K. Sangaiah, M. Pelillo.
Rights and permissions
About this article
Cite this article
Chen, C., Vong, CM., Wong, CM. et al. Efficient extreme learning machine via very sparse random projection. Soft Comput 22, 3563–3574 (2018). https://doi.org/10.1007/s00500-018-3128-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-018-3128-7