Skip to main content
SpringerLink
Log in

Incremental extreme learning machine based on deep feature embedded

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Extreme learning machine (ELM) algorithm is used to train Single-hidden Layer Feed forward Neural Networks. And Deep Belief Network (DBN) is based on Restricted Boltzmann Machine (RBM). The conventional DBN algorithm has some insufficiencies, i.e., Contrastive Divergence (CD) Algorithm is not an ideal approximation method to Maximum Likelihood Estimation. And bad parameters selected in RBM algorithm will produce a bad initialization in DBN model so that we will spend more training time and get a low classification accuracy. To solve the problems above, we summarize the features of extreme learning machine and deep belief networks, and then propose Incremental extreme learning machine based on Deep Feature Embedded algorithm which combines the deep feature extracting ability of Deep Learning Networks with the feature mapping ability of extreme learning machine. Firstly, we introduce Manifold Regularization to our model to attenuate the complexity of probability distribution. Secondly, we introduce the semi-restricted Boltzmann machine (SRBM) to our algorithm, and build a deep belief network based on SRBM. Thirdly, we introduce the thought of incremental feature mapping in ELM to the classifier of DBN model. Finally, we show validity of the algorithm by experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Erhan D, Bengio Y, Courville A et al (2010) Why does unsupervised pre-training help deep learning. J Mach Learn Res 11:625–660

    MATH  MathSciNet  Google Scholar 

  2. Hinton GE, Sejnowski TJ (1983) Optimal perceptual inference. In Proc. CVPR1983. Washington DC, pp 448–453

  3. Hinton GE (2002) Training products of experts by minimizing contrastive divergence. Neural Comput 14(8):1711–1800

    Article  MathSciNet  Google Scholar 

  4. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MATH  MathSciNet  Google Scholar 

  5. Lv Q, Dou Y, Niu X et al (2014) Remote Sensing Image Classification Based on DBN Model. J Comput Res Dev 51(9):1911–1918

    Google Scholar 

  6. Honglak L, Rajesh R, Andrew YN (2011) Unsupervised learning of hierarchical representations with convolutional deep belief networks. Commun ACM 54(10):95–103

    Article  Google Scholar 

  7. Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501

    Article  Google Scholar 

  8. Ding SF, Ma G, Shi ZZ (2014) A novel self-adaptive extreme learning machine based on affinity propagation for radial basis function neural network. Neural Comput Appl 24(7–8):1487–1495

    Article  Google Scholar 

  9. Wang XZ, Shao QY, Qing M et al (2013) Architecture selection for networks trained with extreme learning machine using localized generalization error model. Neurocomputing 102(2):3–9

    Google Scholar 

  10. Huang G, Song S, Gupta JND et al (2014) Semi-supervised and unsupervised extreme learning machines. IEEE Trans Cybern 44(12):2405–2417

    Article  Google Scholar 

  11. He Q, Jin X, Du CY et al (2014) Clustering in extreme learning machine feature space. Neurocomputing 128:88–95

    Article  Google Scholar 

  12. Fu AM, Wang XZ, He YL et al (2014) A study on residence error of training an extreme learning machine and its application to evolutionary algorithms. Neurocomputing 146(1):75–82

    Article  Google Scholar 

  13. Wang XZ, Chen AX, Feng HM (2011) Upper integral network with extreme learning mechanism. Neurocomputing 74(16):2520–2525

    Article  Google Scholar 

  14. Zhang N, Ding SF, Shi ZZ Denoising Laplacian multi-layer extreme learning machine. Neurocomputing (to be published)

  15. Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feed-forward neural networks. In: Proc. IJCNN2004, Budapest, pp 25–29

  16. Osindero S, Hinton GE (2008) Modeling image patches with a directed hierarchy of Markov random fields. Adv Neural Inf Process Syst, pp 1121–1128

  17. Salakhutdinov R (2009) Learning deep generative models. Topics Cogn Sci 3(1):74–91

    Google Scholar 

  18. Salakhutdinov R (2008) Learning and evaluating Boltzmann machines. In: Technical Report UTML TR, Department of Computer Science, University of Toronto

  19. Tieleman T (2008) Training restricted Boltzmann machines using approximations to the likelihood gradient. In: Proc. ICML’08. New York, pp 1064–1071

  20. Tieleman T, Hinton GE (2009) Using fast weights to improve persistent contrastive divergence. In: Proc. ICML’09. New York, pp 1033–1040

  21. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7(3):2399–2434

    MATH  MathSciNet  Google Scholar 

  22. Huang GB, Chen L, Siew CK (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. Neural Netw IEEE Trans 17(4):879–892

    Article  Google Scholar 

  23. Norouzi M, Ranjbar M, Mori G (2009) Stacks of convolutional restricted Boltzmann machines for shift-invariant feature learning. In: Proc. CVPR2009, Miami, FL, pp 2735–2742

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 61379101), and the National Key Basic Research Program of China (No. 2013CB329502).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, 221116, China

    Jian Zhang, Shifei Ding & Nan Zhang

  2. Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China

    Jian Zhang, Shifei Ding, Nan Zhang & Zhongzhi Shi

Authors
  1. Jian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shifei Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhongzhi Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shifei Ding.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Ding, S., Zhang, N. et al. Incremental extreme learning machine based on deep feature embedded. Int. J. Mach. Learn. & Cyber. 7, 111–120 (2016). https://doi.org/10.1007/s13042-015-0419-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-015-0419-5

Keywords

Search

Navigation