Skip to main content
SpringerLink
Log in

Hierarchical extreme learning machine with L21-norm loss and regularization

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Recently, multilayer extreme learning machine (ELM) algorithms have been extensively studied for hierarchical abstract representation learning in the ELM community. In this paper, we investigate the specific combination of \(L_{21}\)-norm based loss function and regularization to improve the robustness and the sparsity of multilayer ELM. As we all known, the mean square error (MSE) cost function (or squared \(L_{2}\)-norm cost function) is commonly used as optimization cost function for ELM, but it is sensitive to outliers and impulsive noises that are pervasive in real-world data. Our \(L_{21}\)-norm loss function can lessen the harmful influence caused by noises and outliers and enhance robustness and stability of the learned model. Additionally, the row sparse inducing \(L_{21}\)-norm regularization can learn the most-relevant sparse representation and reduce the intrinsic complexity of the learning model. We propose a specific combination of \(L_{21}\)-norm loss function and regularization ELM auto-encoder (LR21-ELM-AE), and then stack LR21-ELM-AE hierarchically to construct the hierarchical extreme learning machine (H-LR21-ELM). Experiments conducted on several well-known benchmark datasets are presented, the results show that the proposed H-LR21-ELM can generate a more robust, more discriminative and sparser model compared with the other state-of-the-art multilayer ELM algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig.4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Hinton GE, Salakhutdinov R (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  2. Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. In: Proceedings of advances in nineteenth neural information processing systems, Vancouver, Canada, pp 153–160

  3. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):120–127

    Article  MathSciNet  Google Scholar 

  4. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Article  Google Scholar 

  5. LeCun Y, Bengio Y, Hinton GE (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  6. Ruder S (2016) An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747

  7. Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501

    Article  Google Scholar 

  8. Huang GB, Zhou H, Ding X et al (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B (Cybern) 42(2):513–529

    Article  Google Scholar 

  9. Huang GB (2014) An insight into extreme learning machines: random neurons, random features and kernels. Cogn Comput 6(3):376–390

    Article  Google Scholar 

  10. Huang GB (2015) What are extreme learning machines? Filling the gap between Frank Rosenblatt’s dream and John von Neumann’s puzzle. Cogn Comput 7(3):263–278

    Article  Google Scholar 

  11. Huang GB, Song SJ et al (2015) Trends in extreme learning machines: a review. Neural Netw 61:32–48

    Article  Google Scholar 

  12. Huang ZY, Yu YL, Gu JS et al (2017) An efficient method for traffic sign recognition based on extreme learning machine. IEEE Trans Cybern 47(4):920–933

    Article  Google Scholar 

  13. Wang Y, Wang A, Ai Q et al (2019) Ensemble based fuzzy weighted extreme learning machine for gene expression classification. Appl Intell 49(3):1161–1171

    Article  Google Scholar 

  14. Zhang Y, Wang Y, Zhou G (2018) Multi-kernel extreme learning machine for EEG classification in brain–computer interfaces. Expert Syst Appl 96:302–310

    Article  Google Scholar 

  15. Zhao H, Guo X, Wang M et al (2018) Analyze EEG signals with extreme learning machine based on PMIS feature selection. Int J Mach Learn Cybern 9(2):243–249

    Article  Google Scholar 

  16. Yu Y, Choi TM, Hui CL (2012) An intelligent quick prediction algorithm with applications in industrial control and loading problems. IEEE Trans Autom Sci Eng 9(2):276–287

    Article  Google Scholar 

  17. Zhang Y, Slaughter DC (2011) Hyperspectral species mapping for automatic weed control in tomato under thermal environmental stress. Comput Electron Agric 77(1):95–104

    Article  Google Scholar 

  18. Wang HB, Liu X, Song P et al (2019) Sensitive time series prediction using extreme learning machine. Int J Mach Learn Cybern. https://doi.org/10.1007/s13042-019-00924-7

    Article  Google Scholar 

  19. Kasun LLC, Zhou H, Huang GB et al (2013) Representational learning with extreme learning machine for big data. IEEE Intell Syst 28(6):31–34

    Google Scholar 

  20. Wong CM, Wong PK (2018) Kernel-based multilayer extreme learning machines for representation learning. IEEE Trans Neural Netw Learn Syst 29(3):757–762

    Article  MathSciNet  Google Scholar 

  21. Sun K, Zhang JS, Zhang CX et al (2017) Generalized extreme learning machine autoencoder and a new deep neural network. Neurocomputing 230:374–381

    Article  Google Scholar 

  22. Tang JX, Deng WC, Huang GB (2016) Extreme learning machine for multilayer perceptron. IEEE Trans Netw Learn Syst 27(4):809–821

    Article  MathSciNet  Google Scholar 

  23. NanZ DSF, Shi ZZ (2016) Denoising Laplacian multi-layer extreme learning machine. Neurocomputing 171:1066–1074

    Article  Google Scholar 

  24. Yu WC, Zhuang FZ, He Q (2015) Learning deep representations via extreme learning machines. Neurocomputing 149(Part A):308–315

    Article  Google Scholar 

  25. Yang Y, Jonathan M, Wu QM (2016) Multilayer extreme learning machine with subnetwork nodes for representation learning. IEEE Trans Cybern 46(11):2570–2583

    Article  Google Scholar 

  26. Nie FP, Huang H, Cai X et al (2010) Efficient and robust feature selection via joint L21-norms minimization. In: Proceedings of advances in twenty-third neural information processing systems, Vancouver, Canada, pp 1813–1821

  27. Dong W, Wu XJ (2018) Robust low rank subspace segmentation via joint L21-norm minimization. Neural Process Lett 48(1):99–312

    Article  Google Scholar 

  28. Li R, Wang XD, Lei L et al (2019) L21-norm based loss function and regularization extreme learning machine. IEEE Access 7:6575–6586

    Article  Google Scholar 

  29. Chen LJ, Qu H, Zhao JH et al (2016) Efficient and robust deep learning with correntropy-induced loss function. Neural Comput Appl 27(4):1019–1031

    Article  Google Scholar 

  30. Blake CL, Merz CJ (1998) UCI repository of machine learning databases. In: Department of Information Computer Science, University of California, Irvine, CA. [Online]. http://archive.ics.uci.edu/m

  31. Mike M (1989) Statistical datasets. In: Department of Statistics, University of Carnegie Mellon, Pittsburgh, PA, [Online]. http://lib.stat.cmu.edu/datasets/

  32. Yao L, Ge ZQ (2018) Deep learning of semisupervised process data with hierarchical extreme learning machine and soft sensor application. IEEE Trans Ind Electron 65(2):1490–1498

    Article  Google Scholar 

  33. Chen L, Zhang YH, Huang GH et al (2018) Discriminating cirRNAs from other lncRNAs using a hierarchical extreme learning machine (H-ELM) algorithm with feature selection. Mol Genet Genom 293:137–149

    Article  Google Scholar 

  34. Duan LJ, Bao MH, Cui S et al (2017) Moter imagery classification based on kernel hierarchical extreme learning machine. Cogn Comput 9(6):758–765

    Article  Google Scholar 

  35. He Q, Shang T, Zhuang F, Shi Z (2013) Parallel extreme learning machine for regression based on MapReduce. Neurocomputing 102:52–58

    Article  Google Scholar 

  36. Garea AS, Heras DB, Argüello F (2016) GPU classification of remote-sensing images using kernel ELM and extended morphological profiles. Int J Remote Sens 37:5918–5935

    Article  Google Scholar 

  37. Chen C, Li K, Ouyang A et al (2017) GPU-accelerated parallel hierarchical extreme learning machine on flink for big data. IEEE Trans Syst Man Cybern Syst 47(10):2740–2753

    Article  Google Scholar 

  38. Alaba PA, Popoola SI, Olatomiwa L et al (2019) Towards a more efficient and cost-sensitive extreme learning machine: a state-of-art review of recent trend. Neurocomputing 350:70–90

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China under Grants 61806219, 61876189, 61503407, 61703426, 61273275. This work is also supported by Young Talent fund of University Association for Science and Technology in Shaanxi, China, no. 20190108.

Author information

Authors and Affiliations

  1. College of Air and Missile Defense, Air Force Engineering University, Xi’an, 710051, People’s Republic of China

    Rui Li, Xiaodan Wang, Yafei Song & Lei Lei

Authors
  1. Rui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaodan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yafei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaodan Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, R., Wang, X., Song, Y. et al. Hierarchical extreme learning machine with L21-norm loss and regularization. Int. J. Mach. Learn. & Cyber. 12, 1297–1310 (2021). https://doi.org/10.1007/s13042-020-01234-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01234-z

Keywords

Search

Navigation