Skip to main content
SpringerLink
Log in

Weight Uncertainty in Boltzmann Machine

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Background

Based on restricted Boltzmann machine (RBM), the deep learning models can be roughly divided into deep belief networks (DBNs) and deep Boltzmann machine (DBM). However, the overfitting problems commonly exist in neural networks and RBM models. In order to alleviate the overfitting problem, lots of research has been done. This paper alleviated the overfitting problem in RBM and proposed the weight uncertainty semi-restricted Boltzmann machine (WSRBM) to improve the ability of image recognition and image reconstruction.

Methods

First, this paper built weight uncertainty RBM model based on maximum likelihood estimation. And in the experimental section, this paper verified the effectiveness of the weight uncertainty deep belief network and the weight uncertainty deep Boltzmann machine. Second, in order to obtain better reconstructed images, this paper used the semi-restricted Boltzmann machine (SRBM) as the feature extractor and built the WSRBM. Lastly, this paper used hybrid Monte Carlo sampling and cRBM to improve the classification ability of WSDBM.

Results

The experiments showed that the weight uncertainty RBM, weight uncertainty DBN and weight uncertainty DBM were effective compared with the dropout method. And the WSDBM model performed well in image recognition and image reconstruction as well.

Conclusions

This paper introduced the weight uncertainty method to RBM, and proposed a WSDBM model, which was effective in image recognition and image reconstruction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Erhan D, Vincent P, Bengio Y. Why does unsupervised pre-training help deep learning. J Mach Learn Res. 2010;11(3):625–60.

    Google Scholar 

  2. Hinton G. Training products of experts by minimizing contrastive divergence. Neural Comput. 2002;14(8):1771–800.

    Article  PubMed  Google Scholar 

  3. Roux N, Bengio Y. Representational power of restricted Boltzmann machines and deep belief networks. Neural Comput. 2008;20(6):1631–49.

    Article  PubMed  Google Scholar 

  4. Hinton G, Osindero S, Teh Y. A fast learning algorithm for deep belief nets. Neural Comput. 2006;18(7):1527–54.

    Article  PubMed  Google Scholar 

  5. Hinton G, Salakhutdinov R. Reducing the dimensionality of data with neural networks. Science. 2006;313(5786):504–7.

    Article  CAS  PubMed  Google Scholar 

  6. Lee H, Pham P, Yan L, et al. Unsupervised feature learning for audio classification using convolutional deep belief networks. Adv Neural Inf Process Syst. 2009;22:1096–1104.

    Google Scholar 

  7. Norouzi M, Ranjbar M, Mori G. Stacks of convolutional restricted Boltzmann machines for shift-invariant feature learning. IEEE Conf Comput Vis Pattern Recognit. 2009;1–4:2735–2742.

    Google Scholar 

  8. Salakhutdinov R, Larochelle H. Efficient learning of deep Boltzmann machines. J Mach Learn Res. 2010;9(8):693–700.

    Google Scholar 

  9. Salakhutdinov R, Hinton G. An efficient learning procedure for deep Boltzmann machines. Neural Comput. 2012;24(8):1967–2006.

    Article  PubMed  Google Scholar 

  10. Bengio Y, Boulanger-Lewandowski N, Pascanu R. Advances in optimizing recurrent networks. IEEE International Conference on Acoustics, 2012: 8624–8628.

  11. Zhang J, Ding S, Zhang N, et al. Incremental extreme learning machine based on deep feature embedded. Int J Mach Learn Cybernet. 2016;7(1):111–20.

    Article  CAS  Google Scholar 

  12. Zhang N, Ding S, Shi Z. Denoising Laplacian multi-layer extreme learning machine. Neurocomputing. 2016;171:1066–74.

    Article  Google Scholar 

  13. Ding S, Zhang N, Xu X, et al. Deep Extreme learning machine and its application in EEG classification. Math Probl Eng. 2015;2015:1–11.

    CAS  Google Scholar 

  14. Li J, Li X, Yang B, et al. Segmentation-based image copy-move forgery detection scheme. IEEE Trans Inf Forensics Secur. 2015;10(3):507–18.

    Article  Google Scholar 

  15. Zheng Y, Jeon B, Xu D, et al. Image segmentation by generalized hierarchical fuzzy C-means algorithm. J Intell Fuzzy Syst. 2015;28(2):961–73.

    Google Scholar 

  16. Gu B, Sheng V, Tay K, et al. Incremental support vector learning for ordinal regression. IEEE Trans Neural Netw Learn Syst. 2015;26(7):1403–16.

    Article  PubMed  Google Scholar 

  17. Srivastava N, Hinton G, Krizhevsky A. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15(1):1929–58.

    Google Scholar 

  18. Blundell C, Cornebise J, Kavukcuoglu K. Weight uncertainty in neural networks. International Conference on Machine Learning, France, 2015.

  19. Osindero S, Hinton G. Modeling image patches with a directed hierarchy of markov random fields. Adv Neural Inf Process Syst. 2008;20:1121–1128.

    Google Scholar 

  20. Lee T, Yoon S. Boosted categorical restricted Boltzmann machine for computational prediction of splice junctions. International Conference on Machine Learning, France, 2015.

  21. Hinton G. A practical guide to training restricted Boltzmann machines. Momentum. 2010;9(1):926.

    Google Scholar 

  22. Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images. Technical report U. Toronto, 2009.

  23. Salakhutdinov R. Learning and evaluating Boltzmann machines, Technical Report U. Toronto, 2008

  24. Neal R. Connectionist learning of belief networks. Artif Intell. 1992;56(92):71–113.

    Article  Google Scholar 

  25. Kirkpatrick S, Jr G, Vecchi M. Optimization by simulated annealing. Science. 1983;220(4598):671–80.

    Article  CAS  PubMed  Google Scholar 

  26. Salakhutdinov R. Learning in Markov random fields using tempered transitions. Adv Neural Inf Process Syst. 2009;22:1598–1606.

    Google Scholar 

  27. Desjardins G, Courville A, Bengio Y, et al. Tempered markov chain monte carlo for training of restricted Boltzmann machines. International Conference on Artificial Intelligence and Statistics, Italy, 2010: 145–152.

  28. Tieleman T. Training restricted Boltzmann machines using approximations to the likelihood gradient. International Conference on Machine Learning, Finland, 2008: 1064–1071.

  29. Tieleman T, Hinton G. Using fast weights to improve persistent contrastive divergence. International Conference on Machine Learning, Canada, 2009: 1033–1040.

  30. Jordan M, Ghahramani Z, Jaakkola T, et al. An introduction to variational methods for graphical models. Mach Learn. 1999;37(2):183–233.

    Article  Google Scholar 

  31. Ackley H, Hinton G, Sejnowski J, et al. a mean field theory learning algorithm for neural network. Complex Syst. 1987;1:995–1019.

    Google Scholar 

  32. Hinton G, Salakhutdinov R. A better way to pretrain deep Boltzmann machines. Adv Neural Inf Process Syst. 2012;3:2447–2455.

    Google Scholar 

  33. Ranzato M, Krizhevsky A, Hinton G. Factored 3-way restricted Boltzmann machines for modeling natural images. J Mach Learn Res. 2010;9:621–8.

    Google Scholar 

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 61379101), the National Natural Science Foundation of China (No. 61672522), the National Key Basic Research Program of China (No. 2013CB329502), the Priority Academic Program Development of Jiangsu Higer Education Institutions and the Jiangsu Collaborative Innovation Center on Atmospheric Environment and Equipment Technology.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, 221116, China

    Jian Zhang, Shifei Ding & Nan Zhang

  2. Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China

    Jian Zhang, Shifei Ding & Nan Zhang

  3. College of Computer and Software, Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Yu Xue

Authors
  1. Jian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shifei Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shifei Ding.

Ethics declarations

Conflict of Interest

Jian Zhang, Shifei Ding, Nan Zhang and Yu Xue declare that they have no conflict of interest.

Informed Consent

Informed consent was not required as no human or animals were involved.

Human and Animal Rights

This article does not contain any studies with human or animal subjects performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Ding, S., Zhang, N. et al. Weight Uncertainty in Boltzmann Machine. Cogn Comput 8, 1064–1073 (2016). https://doi.org/10.1007/s12559-016-9429-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-016-9429-1

Keywords

Search

Navigation