Skip to main content
SpringerLink
Log in

Discovering the impact of hidden layer parameters on non-iterative training of feed-forward neural networks

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Considering restricted Boltzmann machine (RBM) as an unsupervised pre-training phase, this paper delivers a study on predetermined model parameters in extreme learning machine (ELM). Because of the non-iterative attribute in fine-tuning phase, the property of hidden layer output plays an important part in model performance. For ELM, we give a theoretical analysis on the hidden layer parameters related to matrix perturbation and continuity of generalized inverse. Then by empirically analyzing the proposed RBM–ELM algorithm, we find that the impact of hidden layer parameters on generalization ability varies among the experimental datasets. By exploring the training process and comparing the model parameters between random assignment and RBM, we identify the special pattern of hidden layer output discussed in theoretical part and empirically show that such pattern could harm the model performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. In: Advances in neural information processing systems, pp 153–160

  • Chen L, Yang L, Sun C, Xi H (2017) A fast RBM-hidden-nodes based extreme learning machine. In: Control And Decision Conference (CCDC), 2017 29th Chinese, IEEE, pp 2121–2126

  • Ding L, Han B, Wang S, Li X, Song B (2017a) User-centered recommendation using us-elm based on dynamic graph model in e-commerce. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0751-z

  • Ding S, Zhang N, Zhang J, Xu X, Shi Z (2017b) Unsupervised extreme learning machine with representational features. Int J Mach Learn Cybernet 8(2):587–595

    Article  Google Scholar 

  • Erhan D, Manzagol PA, Bengio Y, Bengio S, Vincent P (2009) The difficulty of training deep architectures and the effect of unsupervised pre-training. In: Artificial Intelligence and Statistics, pp 153–160

  • Erhan D, Bengio Y, Courville A, Manzagol PA, Vincent P, Bengio S (2010) Why does unsupervised pre-training help deep learning? J Mach Learn Res 11:625–660

    MathSciNet  MATH  Google Scholar 

  • Fu AM, Wang XZ, He YL, Wang LS (2014) A study on residence error of training an extreme learning machine and its application to evolutionary algorithms. Neurocomputing 146:75–82

    Article  Google Scholar 

  • Hinton GE (2006) Training products of experts by minimizing contrastive divergence. Neural Comput 14(8):1771–1800

    Article  MATH  Google Scholar 

  • Hinton G (2010) A practical guide to training restricted boltzmann machines. Momentum 9(1):926

    Google Scholar 

  • Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  • Hinton G, Deng L, Yu D, Dahl GE, Ar Mohamed, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Article  Google Scholar 

  • Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks, 2004. Proceedings. IEEE, vol 2, pp 985–990

  • Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501

    Article  Google Scholar 

  • Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybernet Part B (Cybernet) 42(2):513–529

    Article  Google Scholar 

  • LeCun Y, Cortes C, Burges CJ (2010) Mnist handwritten digit database. AT&T Labs Available http://yann.lecun.com/exdb/mnist/

  • Li F, Liu H, Xu X, Sun F (2017) Haptic recognition using hierarchical extreme learning machine with local-receptive-field. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0736-y

  • Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml

  • Mao W, Wang J, Xue Z (2017) An elm-based model with sparse-weighting strategy for sequential data imbalance problem. Int J Mach Learn Cybernet 8(4):1333–1345

    Article  Google Scholar 

  • Meng L, Ding S, Xue Y (2017) Research on denoising sparse autoencoder. Int J Mach Learn Cybernet 8(5):1719–1729

    Article  Google Scholar 

  • Pacheco A, Krohling R, da Silva C (2017) Restricted boltzmann machine to determine the input weights for extreme learning machines. arXiv preprint arXiv:1708.05376

  • Parlett BN (1998) The symmetric eigenvalue problem. SIAM, Philadelphia

    Book  MATH  Google Scholar 

  • Salakhutdinov R, Mnih A, Hinton G (2007) Restricted boltzmann machines for collaborative filtering. In: Proceedings of the 24th international conference on machine learning. ACM, pp 791–798

  • Smolensky P (1986) Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado University at Boulder Department of Computer Science

  • Wang R, Kwong S, Wang X (2012) A study on random weights between input and hidden layers in extreme learning machine. Soft Comput 16(9):1465–1475

    Article  Google Scholar 

  • Wang R, He YL, Chow CY, Ou FF, Zhang J (2015) Learning elm-tree from big data based on uncertainty reduction. Fuzzy Sets Syst 258:79–100

    Article  MathSciNet  MATH  Google Scholar 

  • Wang R, Chow CY, Lyu Y, Lee V, Kwong S, Li Y, Zeng J (2017a) Taxirec: recommending road clusters to taxi drivers using ranking-based extreme learning machines. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2017.2772907

  • Wang R, Xie H, Feng J, Wang FL, Xu C (2017b) Multi-criteria decision making based architecture selection for single-hidden layer feedforward neural networks. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0746-9

  • Wang XZ, Zhang T, Wang R (2017c) Noniterative deep learning: incorporating restricted boltzmann machine into multilayer random weight neural networks. IEEE Trans Syst Man Cybernet Syst. https://doi.org/10.1109/TSMC.2017.2701419

  • Wang XZ, Wang R, Xu C (2018) Discovering the relationship between generalization and uncertainty by incorporating complexity of classification. IEEE Trans Cybernet 48(2):703–715

    Article  Google Scholar 

  • Yu D, Deng L, Dahl G (2010) Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition. In: Proceedings of NIPS workshop on deep learning and unsupervised feature learning

  • Zhai J, Zhang S, Wang C (2017) The classification of imbalanced large data sets based on mapreduce and ensemble of elm classifiers. Int J Mach Learn Cybernet 8(3):1009–1017

    Article  Google Scholar 

  • Zhang H, Zhang S, Yin Y, Chen X (2017) Prediction of the hot metal silicon content in blast furnace based on extreme learning machine. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0674-8

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Grants 61772344 and 61732011), in part by the Natural Science Foundation of SZU (Grants 827-000140, 827-000230, and 2017060), in part by the Youth Foundation Project of Hebei Natural Science Foundation of China (F2018511002), in part by Macao Science and Technology Development Funds (100/2013/A3& 081/2015/A3), and in part by the Interdisciplinary Innovation Team of Shenzhen University.

Author information

Authors and Affiliations

  1. Guangdong Key Laboratory of Intelligent Information Processing, College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China

    Zhiqi Huang

  2. College of Mathematics and Statistics, Shenzhen University, Shenzhen, China

    Ran Wang

  3. Faculty of Information Technology, Macau University of Science and Technology, Macao, China

    Hong Zhu

  4. Department of Information Management, Central Institute for Correctional Police, Baoding, Hebei, China

    Jie Zhu

Authors
  1. Zhiqi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ran Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ran Wang.

Ethics declarations

Conflict of interest

All the authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Communicated by X. Wang, A.K. Sangaiah, M. Pelillo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, Z., Wang, R., Zhu, H. et al. Discovering the impact of hidden layer parameters on non-iterative training of feed-forward neural networks. Soft Comput 22, 3495–3506 (2018). https://doi.org/10.1007/s00500-018-3022-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-018-3022-3

Keywords

Search

Navigation