Skip to main content
Springer Nature Link
Log in

A precise method for RBMs training using phased curricula

  • 1199: Computational Intelligence Revolution in Multimedia Data Analytics and Business Management
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Restricted Boltzmann machines (RBMs) are efficacious undirected neural networks for generating features and reconstructing images. Nevertheless, the classical persistent chain sampling algorithm has the problem of refactoring failure in the early training stage, which significantly limits the feature extraction and application of RBM. In this paper, motivated by the cumulative nature of the curriculum learning, three Phased Gibbs Sampling (PGS) methods are proposed for more efficient feature extraction and reconstruction by training the RBM periodically. Then, to achieve an automatic and exclusive training step, the innovative Improved Dynamic Learning Rate (IDLR) is designed by cooperating with the reconstruction error and the anti-vibration coefficient. Extensive experimental results of MNIST, 20 Newsgroup, Olivetti face, MNORB, and USPS demonstrate the superiority of three PGS-IDLR algorithms in terms of reconstruction error, training time, and classification accuracy. More specifically, the proposed algorithms can improve the classification accuracy by at least 2% and shorten the training time, compared with the state-of-the-art approaches. Moreover, they achieve a better performance in log-likelihood indictor and image reconstruction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

References

  1. Bengio Y (2009) Learning deep architectures for ai. Foundations and trends?. Mach Learn 2:1–127

    MATH  Google Scholar 

  2. Bengio Y, Louradour J, Collobert R, Weston J (2009) Curriculum learning, the 26th Annual International Conference on Machine Learning, pp 41–48

  3. Brakel P, Dieleman S, Schrauwen B (2012) Training restricted Boltzmann machines with multi-tempering: Harnessing parallelization. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, pp 287–292

  4. Cai D, He X, Han J, Zhang H (2006) Orthogonal laplacianfaces for face recognition. IEEE Trans Image Process 15:3608–3614

  5. Cho K (2011) Enhanced gradient and adaptive learning rate for training restricted boltzmann machines. Proceedings of the 28th International Conference on Machine Learning

  6. Cho K, Raiko T, Ilin A (2010) Parallel tempering is efficient for learning restricted Boltzmann machines. Int Joint Conf Neural Netw:3246–3253

  7. Coates A, Ng AY, Lee H (2010) An analysis of single-layer networks in unsupervised feature learning. International Conference on Artificial Intelligence and Statistics

  8. Desjardins G, Courville A, Bengio Y (2010) Adaptive parallel tempering for stochastic maximum likelihood learning of RBMs. Advances in Neural Information Processing Systems Workshop on Deep Learning and Unsupervised Feature Learning

  9. EHinton GE (2002) Training products of experts by minimizing contrastive divergence. Neural Comput 14:1771–1800

    Article  MATH  Google Scholar 

  10. EHinton GE (2006) Reducing the dimensionality of data with neural networks. Science 313:504–507

    Article  MathSciNet  Google Scholar 

  11. Elman JL (1993) Learning and development in neural networks: the importance of starting small. Cognition 48:71–99

    Article  Google Scholar 

  12. Fakhari A, Kiani K (2021) A new restricted boltzmann machine training algorithm for image restoration, Multimed Tools Appl 80, 2047–2062

  13. Fatemi M, Safayani M (2019) Joint, sentiment/topic modeling on text data using a boosted restricted Boltzmann Machine. Multimed Tools Appl 78:20637–20653

  14. Fischer A, Igel C (2011) Parallel tempering, importance sampling, and restricted Boltzmann machines. The 5th Workshop on Theory of Randomized Search Heuristics

  15. Fischer A, Igel C (2014) Training restricted Boltzmann machines: An introduction. Patern Recogn 47:25–39

    Article  MATH  Google Scholar 

  16. Hinton G (2012) A practical guide to training restricted Boltzmann machines. Momentum 9:599–619

    Google Scholar 

  17. Hinton G, Salakhutdinov R (2009) Replicated softmax: an undirected topic model. NIPS, pp 1607–1614

  18. Hinton G, Osindero S, Teh YW (2016) A fast learning algorithm for deep belief nets. Foundations and Trends?. Neural Comput 18:1527–1554

    Article  MATH  Google Scholar 

  19. Hinton G, Deng L, Yu D, Dahl GE, Mohamed A, Jaitly N (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. Signal Process Mag IEEE 29:82–97

    Article  Google Scholar 

  20. Hongmei W, Pengzhong L (2021) Image recognition based on improved convolutional deep belief network model. Multimed Tools Appl 80:2031–2045

  21. Jiang L, Meng D, Zhao Q, Shan S, Hauptmann AG (2015) Self-paced curriculum learning. Twenty-Ninth AAAI Conference on Artificial Intelligence

  22. Koller D, Friedman N (2009) Probabilistic graphical models: Principles and Techniques. MIT Press

  23. Krueger KA, Dayan P (2009) Flexible shaping: How learning in small steps helps. Cognition 110:380–394

    Article  Google Scholar 

  24. Kumar MP, Packer B, Koller D (2010) Self-paced learning for latent variable models. Advances in Neural Information Processing Systems

  25. Lang K (1995) Newsweeder: Learning to filter netnews. Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, pp 331–339

  26. Larochelle H, Bengio Y (2008) Classification using discriminative restricted Boltzmann machines. Machine Learning Proceedings of the Twenty-Fifth International Conference

  27. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436

    Article  Google Scholar 

  28. Lėcun Y, Bottou L, Bengio Y, et al. (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324

    Article  Google Scholar 

  29. Lėcun Y, Huang FJ, Bottou L (2004) Learning methods for generic object recognition with invariance to pose and lighting: Computer Vision and Pattern Recognition. Proceedings of the 2004 IEEE Computer Society Conference

  30. Li F, Gao X, Wan K (2016) Research on RBM training algorithm based on dynamic Gibbs sampling. Acta Autom Sin 42:931–942

    Google Scholar 

  31. Li F, Gao X, Wan K (2017) Research on RBM networks training based on improved parallel tempering algorithm. Acta Autom Sin 43:753–764

    MATH  Google Scholar 

  32. Luo L, Wang Y, Peng H, Tang Z, You S, Huang X (2016) Training restricted Boltzmann Machine with dynamic learning rate. International Conference on Computer Science and Education. IEEE

  33. Medhat F, Chesmore D, Robinson J (2017) Recognition of acoustic events using masked conditional neural networks. 16th IEEE International Conference on Machine Learning and Applications

  34. Neal RM (1992) Connectionist learning of deep belief networks. Artif Intell 56:71–113

    Article  MATH  Google Scholar 

  35. Rohde D, Plaut D (1999) Language acquisition in the absence of explicit negative evidence: How important is starting small. Cognition 72:67–109

    Article  Google Scholar 

  36. Sailor HB, Patil HA (2016) Unsupervised learning of temporal receptive fields using convolutional RBM for ASR task. IEEE 24th European Signal Processing conference(EUSIPCO)

  37. Salakhutdinov R (2015) Learning deep generative models. Ann Rev Stat Appl 2:361–385

    Article  Google Scholar 

  38. Salakhutdinov R, Hinton G (2009) Deep boltzmann machines. J Mach Learn Res 5:1967–2006

    MATH  Google Scholar 

  39. Salakhutdinov R, Murray I (2008) On the quantitative analysis of deep belief networks. International Conference on Machine Learning

  40. Sanger TD (1994) Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE Trans Robot Autom 10:323–333

    Article  Google Scholar 

  41. Schwenk H, Gauvain J. -L. (2002) Connectionist language modeling for large vocabulary continuous speech recognition. International Conference on Acoustics Speech and Signal Processing

  42. Spitkovsky VI, Alshawi H, Jurafsky D (2010) From baby steps to leapfrog: how “less is more” in unsupervised dependency parsing. NAACL

  43. Swersky K, Bo C, Marlin B, Freitas ND (2010) A tutorial on stochastic approximation algorithms for training. Restricted Boltzmann Machines and Deep Belief Nets Information Theory and Applications Workshop

  44. Tang Y, Salakhutdinov R, Hinton G (2012) Robust boltzmann machines for recognition and denoising. In: 2012 IEEE Conference on computer vision and pattern recognition, pp 2264–2271

  45. Tehrani AA, Nickfarjam AM, Ebrahimpour-komleh H et al (2021) Multi-input 2-dimensional deep belief network: diabetic retinopathy grading as case study. Multimed Tools Appl 80:6171–6186

  46. Tieleman T (2008) Training restricted Boltzmann machines using approximations to the likelihood gradient. Int Conf Mach Learn:1064–1071

  47. Tieleman T, Hinton G (2009) Using fast weights to improve persistent contrastive divergence. International Conference on Machine Learning

  48. Wang Q, Gao X, Wan K, Li F, Hu Z (2020) A novel restricted boltzmann machine training algorithm with fast gibbs sampling policy. Mathematical Problems in Engineering

  49. Wu Q (2020) Image retrieval method based on deep learning semantic feature extraction and regularization softmax. Multimed Tools Appl, 79, 9419–9433

  50. Zhao Y, Chen Y, Tu K, Tian J (2017) Learning bayesian network structures under incremental construction curricula. Neurocomputing 258:30–40

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful to Professor Gao for the discussions on this topic. This study was supported by the National Natural Science Foundation of China (Grant no. 61573285) and Natural Science Foundation of Shaanxi Province (Grant no. 2020JQ-220).

Author information

Authors and Affiliations

  1. Correspondence address:School of Electronic and Information, Northwestern Polytechnical University, Xi’an, Shaanxi, 710129, China

    Qianglong Wang, Xiaoguang Gao, Xinyu Li, Zijian Hu & Kaifang Wan

Authors
  1. Qianglong Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Xiaoguang Gao
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xinyu Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zijian Hu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Kaifang Wan
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Qianglong Wang.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Q., Gao, X., Li, X. et al. A precise method for RBMs training using phased curricula. Multimed Tools Appl 82, 8013–8047 (2023). https://doi.org/10.1007/s11042-022-12973-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12973-2

Keywords