Skip to main content
SpringerLink
Log in

Unsupervised feature learning with sparse Bayesian auto-encoding based extreme learning machine

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Extreme learning machine (ELM) is a popular method in machine learning with extremely few parameters, fast learning speed and model efficiency. Unsupervised feature learning based ELM receives rising research focus. Recently the ELM auto-encoder (ELM-AE) was proposed for this task, which develops the ELM based compact feature learning without sacrificing elegant solution. Compared with ELM-AE and following \(\ell _1\)-regularized ELM-AE, we introduce a sparse Bayesian learning scheme into ELM-AE for better generalization capability. A parallel training strategy is also integrated to improve time-efficiency of multi-output sparse Bayesian learning. Furthermore, pruning hidden nodes for better performance and efficiency according to estimated variances of prior distribution of output weights is achieved. Experiments on several datasets verify the effectiveness and efficiency of our proposed ELM-AE for unsupervised feature learning, compared with PCA, NMF, ELM-AE and \(\ell _1\)-regularized ELM-AE.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://www.numpy.org/devdocs/user/theory.broadcasting.html.

  2. http://www.openml.org.

References

  1. Rumelhart DE, Hinton GE, Williams RJ et al (1988) Learning representations by back-propagating errors. Cognit Model 5(3):1

    MATH  Google Scholar 

  2. Huang G-B, Chen L, Siew CK et al (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892

    Google Scholar 

  3. Huang G-B, Zhou H, Ding X, Zhang R (2011) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B (Cybernetics) 42(2):513–529

    Google Scholar 

  4. Huang G-B, Zhu Q-Y, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501

    Google Scholar 

  5. Nizar AH, Dong ZY, Wang Y (2008) Power utility nontechnical loss analysis with extreme learning machine method. IEEE Trans Power Syst 23(3):946–955

    Google Scholar 

  6. You ZH, Lei YK, Zhu L, Xia J, Wang Bing (2013) Prediction of protein-protein interactions from amino acid sequences with ensemble extreme learning machines and principal component analysis. Bmc Bioinform 14(S8):S10

    Google Scholar 

  7. Zhige Xie X, Kai SW, Liu L, Xiong Y, Hui Huang (2015) Projective feature learning for 3D shapes with multi-view depth images. Comput Graphics Forum 34(7):1–11

    Google Scholar 

  8. Huang G, Song S, Gupta JND, Cheng W (2014) Semi-supervised and unsupervised extreme learning machines. IEEE Trans Cybern 44(12):2405–2417

    Google Scholar 

  9. Kasun LLC, Zhou H, Huang G-B, Vong CM (2013) Representational learning with extreme learning machine for big data. IEEE Intell Syst 28(6):31–34

    Google Scholar 

  10. Kasun LLC, Yang Y, Huang G-B, Zhang Z (2016) Dimension reduction with extreme learning machine. IEEE Trans Image Process 25(8):3906–3918

    MathSciNet  MATH  Google Scholar 

  11. Liyanaarachchi LCC, Tianchi L, Yan Y, Zhiping L, Guang-Bin H (2015) Extreme learning machine for clustering. In: Proceedings of ELM-2014, volume 1. Springer, pp 435–444

  12. Tang J, Deng C, Huang G-B (2015) Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst 27(4):809–821

    MathSciNet  Google Scholar 

  13. Luo J, Vong C-M, Wong P-K (2013) Sparse bayesian extreme learning machine for multi-classification. IEEE Trans Neural Netw Learn Syst 25(4):836–843

    Google Scholar 

  14. Hoerl AE, Kennard W (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1):55–67

    MATH  Google Scholar 

  15. Huang G-B (2014) An insight into extreme learning machines: random neurons, random features and kernels. Cognit Comput 6(3):376–390

    MathSciNet  Google Scholar 

  16. Ding C, He X (2004) K-means clustering via principal component analysis. In: International Conference on machine learning

  17. Pearson K (1901) Liii. on lines and planes of closest fit to systems of points in space. Lond Edinb Dublin Philos Mag J Sci 2(11):559–572

    MATH  Google Scholar 

  18. Hotelling H (1933) Analysis of a complex of statistical variables into principal components. J Educ Psychol 24(6):417

    MATH  Google Scholar 

  19. Yoo Y, Oh S-Y (2016) Fast training of convolutional neural network classifiers through extreme learning machines. In: 2016 International Joint Conference on neural networks (IJCNN). IEEE, pp 1702–1708

  20. Wang Y, Xie Z, Kai X, Dou Y, Lei Yuanwu (2016) An efficient and effective convolutional auto-encoder extreme learning machine network for 3d feature learning. Neurocomputing 174:988–998

    Google Scholar 

  21. Low C-Y, Teoh AB-J (2017) Stacking-based deep neural network: Deep analytic network on convolutional spectral histogram features. In: 2017 IEEE International Conference on Image Processing (ICIP). IEEE, pp 1592–1596

  22. Congdon P (2007) Bayesian statistical modelling, vol 704. Wiley, Hoboken

    MATH  Google Scholar 

  23. Soria-Olivas E, Gomez-Sanchis J, Martin JD, Vila-Frances J, Martinez M, Magdalena JR, Serrano AJ (2011) Belm: Bayesian extreme learning machine. IEEE Trans Neural Netw 22(3):505–509

    Google Scholar 

  24. Bishop CM (2006) Pattern recognition and machine learning. Springer, Berlin

    MATH  Google Scholar 

  25. MacKay DJC (1992) The evidence framework applied to classification networks. Neural Comput 4(5):720–736

    Google Scholar 

  26. Nabney IT (2004) Efficient training of RBF networks for classification. Int J Neural Syst 14(3):1–8

    Google Scholar 

  27. Tipping ME (2001) Sparse Bayesian learning and the relevance vector machine. J Mach Learn Res 1(Jun):211–244

    MathSciNet  MATH  Google Scholar 

  28. MacKay DJC (1992) Bayesian interpolation. Neural Comput 4(3):415–447

    MATH  Google Scholar 

  29. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M et al (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467

  30. Lee DD, Sebastian Seung H (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788

    MATH  Google Scholar 

  31. Michael L, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE international conference on automatic face and gesture recognition, pp 200–205. IEEE

  32. Samaria FS, Harter AC (1994) Parameterisation of a stochastic model for human face identification. In: Proceedings of 1994 IEEE Workshop on Applications of Computer Vision. IEEE, pp 138–142

  33. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on computer vision and pattern recognition

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Electronic Engineering, Nanyang Technological University, Nanyang Avenue, Singapore, 639798, Singapore

    Guanghao Zhang, Dongshun Cui, Shangbo Mao & Guang-Bin Huang

  2. Energy Research Institute @ NTU (ERI@N), Interdisciplinary Graduate School, Nanyang Technological University, Nanyang Avenue, Singapore, 639798, Singapore

    Dongshun Cui

Authors
  1. Guanghao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dongshun Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shangbo Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guang-Bin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanghao Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, G., Cui, D., Mao, S. et al. Unsupervised feature learning with sparse Bayesian auto-encoding based extreme learning machine. Int. J. Mach. Learn. & Cyber. 11, 1557–1569 (2020). https://doi.org/10.1007/s13042-019-01057-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-019-01057-7

Keywords

Search

Navigation