Abstract
Considering restricted Boltzmann machine (RBM) as an unsupervised pre-training phase, this paper delivers a study on predetermined model parameters in extreme learning machine (ELM). Because of the non-iterative attribute in fine-tuning phase, the property of hidden layer output plays an important part in model performance. For ELM, we give a theoretical analysis on the hidden layer parameters related to matrix perturbation and continuity of generalized inverse. Then by empirically analyzing the proposed RBM–ELM algorithm, we find that the impact of hidden layer parameters on generalization ability varies among the experimental datasets. By exploring the training process and comparing the model parameters between random assignment and RBM, we identify the special pattern of hidden layer output discussed in theoretical part and empirically show that such pattern could harm the model performance.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. In: Advances in neural information processing systems, pp 153–160
Chen L, Yang L, Sun C, Xi H (2017) A fast RBM-hidden-nodes based extreme learning machine. In: Control And Decision Conference (CCDC), 2017 29th Chinese, IEEE, pp 2121–2126
Ding L, Han B, Wang S, Li X, Song B (2017a) User-centered recommendation using us-elm based on dynamic graph model in e-commerce. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0751-z
Ding S, Zhang N, Zhang J, Xu X, Shi Z (2017b) Unsupervised extreme learning machine with representational features. Int J Mach Learn Cybernet 8(2):587–595
Erhan D, Manzagol PA, Bengio Y, Bengio S, Vincent P (2009) The difficulty of training deep architectures and the effect of unsupervised pre-training. In: Artificial Intelligence and Statistics, pp 153–160
Erhan D, Bengio Y, Courville A, Manzagol PA, Vincent P, Bengio S (2010) Why does unsupervised pre-training help deep learning? J Mach Learn Res 11:625–660
Fu AM, Wang XZ, He YL, Wang LS (2014) A study on residence error of training an extreme learning machine and its application to evolutionary algorithms. Neurocomputing 146:75–82
Hinton GE (2006) Training products of experts by minimizing contrastive divergence. Neural Comput 14(8):1771–1800
Hinton G (2010) A practical guide to training restricted boltzmann machines. Momentum 9(1):926
Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554
Hinton G, Deng L, Yu D, Dahl GE, Ar Mohamed, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97
Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks, 2004. Proceedings. IEEE, vol 2, pp 985–990
Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501
Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybernet Part B (Cybernet) 42(2):513–529
LeCun Y, Cortes C, Burges CJ (2010) Mnist handwritten digit database. AT&T Labs Available http://yann.lecun.com/exdb/mnist/
Li F, Liu H, Xu X, Sun F (2017) Haptic recognition using hierarchical extreme learning machine with local-receptive-field. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0736-y
Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml
Mao W, Wang J, Xue Z (2017) An elm-based model with sparse-weighting strategy for sequential data imbalance problem. Int J Mach Learn Cybernet 8(4):1333–1345
Meng L, Ding S, Xue Y (2017) Research on denoising sparse autoencoder. Int J Mach Learn Cybernet 8(5):1719–1729
Pacheco A, Krohling R, da Silva C (2017) Restricted boltzmann machine to determine the input weights for extreme learning machines. arXiv preprint arXiv:1708.05376
Parlett BN (1998) The symmetric eigenvalue problem. SIAM, Philadelphia
Salakhutdinov R, Mnih A, Hinton G (2007) Restricted boltzmann machines for collaborative filtering. In: Proceedings of the 24th international conference on machine learning. ACM, pp 791–798
Smolensky P (1986) Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado University at Boulder Department of Computer Science
Wang R, Kwong S, Wang X (2012) A study on random weights between input and hidden layers in extreme learning machine. Soft Comput 16(9):1465–1475
Wang R, He YL, Chow CY, Ou FF, Zhang J (2015) Learning elm-tree from big data based on uncertainty reduction. Fuzzy Sets Syst 258:79–100
Wang R, Chow CY, Lyu Y, Lee V, Kwong S, Li Y, Zeng J (2017a) Taxirec: recommending road clusters to taxi drivers using ranking-based extreme learning machines. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2017.2772907
Wang R, Xie H, Feng J, Wang FL, Xu C (2017b) Multi-criteria decision making based architecture selection for single-hidden layer feedforward neural networks. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0746-9
Wang XZ, Zhang T, Wang R (2017c) Noniterative deep learning: incorporating restricted boltzmann machine into multilayer random weight neural networks. IEEE Trans Syst Man Cybernet Syst. https://doi.org/10.1109/TSMC.2017.2701419
Wang XZ, Wang R, Xu C (2018) Discovering the relationship between generalization and uncertainty by incorporating complexity of classification. IEEE Trans Cybernet 48(2):703–715
Yu D, Deng L, Dahl G (2010) Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition. In: Proceedings of NIPS workshop on deep learning and unsupervised feature learning
Zhai J, Zhang S, Wang C (2017) The classification of imbalanced large data sets based on mapreduce and ensemble of elm classifiers. Int J Mach Learn Cybernet 8(3):1009–1017
Zhang H, Zhang S, Yin Y, Chen X (2017) Prediction of the hot metal silicon content in blast furnace based on extreme learning machine. Int J Mach Learn Cybernet. https://doi.org/10.1007/s13042-017-0674-8
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (Grants 61772344 and 61732011), in part by the Natural Science Foundation of SZU (Grants 827-000140, 827-000230, and 2017060), in part by the Youth Foundation Project of Hebei Natural Science Foundation of China (F2018511002), in part by Macao Science and Technology Development Funds (100/2013/A3& 081/2015/A3), and in part by the Interdisciplinary Innovation Team of Shenzhen University.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All the authors declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Communicated by X. Wang, A.K. Sangaiah, M. Pelillo.
Rights and permissions
About this article
Cite this article
Huang, Z., Wang, R., Zhu, H. et al. Discovering the impact of hidden layer parameters on non-iterative training of feed-forward neural networks. Soft Comput 22, 3495–3506 (2018). https://doi.org/10.1007/s00500-018-3022-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-018-3022-3