Skip to main content
SpringerLink
Log in

Classification model of restricted Boltzmann machine based on reconstruction error

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Many models are used to solve classification problems in machine learning. The classification restricted Boltzmann machine (ClassRBM) is a type of self-contained network model that is widely used in various classification applications. To implement classification, the ClassRBM updates the model parameters constantly during the training phase in terms of their class labels so that the model parameters learned from the ClassRBM are different from those learned from the conventional restricted Boltzmann machine (RBM), which is trained by unsupervised learning. In this paper, we demonstrate that the reconstruction errors of the ClassRBM are larger than those of the conventional RBM because of the label information. We then propose a classification model of the restricted Boltzmann machine based on these reconstruction errors. The reconstruction errors are used to train the proposed model to improve the classification performance of the ClassRBM. Extensive experiments are carried out to verify the proposed model. The experimental results demonstrate that the proposed model can improve the classification performance of the ClassRBM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127

    Article  MATH  Google Scholar 

  2. Bengio Y, Lamblin P, Popovici D, Larochelle H et al (2007) Greedy layer-wise training of deep networks. Adv Neural Inf Process Syst 19:153–160

    Google Scholar 

  3. Blum C, Socha K (2005) Training feed-forward neural networks with ant colony optimization: an application to pattern classification. In: Fifth International Conference on hybrid intelligent systems, 2005. HIS’05, p 6. IEEE

  4. Chen K, Yang L, Yu X, Chi H (1997) A self-generating modular neural network architecture for supervised learning. Neurocomputing 16(1):33–48

    Article  Google Scholar 

  5. Decoste D, Schölkopf B (2002) Training invariant support vector machines. Mach Learn 46(1–3):161–190

    Article  MATH  Google Scholar 

  6. Eiamkanitchat N, Theera-Umpon N, Auephanwiriyakul S (2010) Colon tumor microarray classification using neural network with feature selection and rule-based classification. In: Advances in neural network research and applications. Springer, Berlin, pp 363–372

  7. Elfwing S, Uchibe E, Doya K (2015) Expected energy-based restricted Boltzmann machine for classification. Neural Netw 64:29–38

    Article  MATH  Google Scholar 

  8. Fischer A, Igel C (2012) An introduction to restricted Boltzmann machines. In: Progress in pattern recognition, image analysis, computer vision, and applications. Springer, Berlin, pp 14–36

  9. Fischer A, Igel C (2014) Training restricted Boltzmann machines: an introduction. Pattern Recogn 47(1):25–39

    Article  MATH  Google Scholar 

  10. Freund Y, Haussler D (1992) Unsupervised learning of distributions on binary vectors using two layer networks. In: Advances in neural information processing systems, pp 912–919

  11. Hansen LK, Salamon P (1990) Neural network ensembles. IEEE Trans Pattern Anal Mach Intell 10:993–1001

    Article  Google Scholar 

  12. Hinton GE (2002) Training products of experts by minimizing contrastive divergence. Neural Comput 14(8):1771–1800

    Article  MATH  Google Scholar 

  13. Hinton GE (2007) To recognize shapes, first learn to generate images. Prog Brain Res 165:535–547

    Article  Google Scholar 

  14. Hinton GE (2010) A practical guide to training restricted Boltzmann machines. Momentum 9(1):599–619

    Google Scholar 

  15. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  16. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  MATH  Google Scholar 

  17. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  18. Huang W, Hong H, Bian K, Zhou X, Song G, Xie K (2015) Improving deep neural network ensembles using reconstruction error. In: 2015 International joint conference on neural networks (IJCNN), pp 1–7. IEEE

  19. Huang W, Zhang N, Hu W, Hong H, Song G, Xie K (2014) Dynamic boosting in deep learning using reconstruction error. In: 2014 International joint conference on neural networks (IJCNN), pp 473–480. IEEE

  20. Ji N, Zhang J, Zhang C, Wang L (2014) Discriminative restricted Boltzmann machine for invariant pattern recognition with linear transformations. Pattern Recogn Lett 45:172–180

    Article  Google Scholar 

  21. Ji NN, Zhang JS, Zhang CX (2014) A sparse-response deep belief network based on rate distortion theory. Pattern Recogn 47(9):3179–3191

    Article  Google Scholar 

  22. Koziol JA, Tan EM, Dai L, Ren P, Zhang JY (2014) Restricted Boltzmann machines for classification of hepatocellular carcinoma. Comput Biol J 2014:1–5

    Article  Google Scholar 

  23. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  24. Larochelle H, Bengio Y (2008) Classification using discriminative restricted Boltzmann machines. In: Proceedings of the 25th international conference on Machine learning, pp 536–543. ACM

  25. Larochelle H, Erhan D, Vincent P (2009) Deep learning using robust interdependent codes. In: International conference on artificial intelligence and statistics, pp 312–319

  26. Larochelle H, Mandel M, Pascanu R, Bengio Y (2012) Learning algorithms for the classification restricted Boltzmann machine. J Mach Learn Res 13(1):643–669

    MathSciNet  MATH  Google Scholar 

  27. Le Cun BB, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1990) Handwritten digit recognition with a back-propagation network. In: Advances in neural information processing systems. Citeseer, pp 396–404

  28. Lei Y, He Z, Zi Y, Hu Q (2006) Mechanical fault diagnosis model based on feature evaluation and neural networks. Xi’an Jiaotong Daxue Xuebao (Journal of Xi’an Jiaotong University) 40(5):558–562

    Google Scholar 

  29. Li Q, Zhang J, Wang Y, Kang K (2014) Credit risk classification using discriminative restricted Boltzmann machines. In: 2014 IEEE 17th International conference on computational science and engineering (CSE), pp 1697–1700. IEEE

  30. Liu Z, Pu J, Huang T, Qiu Y (2013) A novel classification method for palmprint recognition based on reconstruction error and normalized distance. Appl Intell 39(2):307–314

    Article  Google Scholar 

  31. Lu J, Ho DW, Cao J, Kurths J (2011) Exponential synchronization of linearly coupled neural networks with impulsive disturbances. IEEE Trans Neural Netw 22(2):329–336

    Article  Google Scholar 

  32. Lv JC, Tan KK, Yi Z, Huang S (2008) Stability and chaos of a class of learning algorithms for ica neural networks. Neural Process Lett 28(1):35–47

    Article  Google Scholar 

  33. Lv JC, Yi Z, Li Y (2015) Non-divergence of stochastic discrete time algorithms for pca neural networks. IEEE Trans Neural Netw Learn Syst 26(2):394–399

    Article  MathSciNet  Google Scholar 

  34. Pang Z, Liu D, Lloyd SR (2007) Classification of obstructive sleep apnea by neural networks. In: Advances in neural networks–ISNN 2007. Springer, Berlin, pp 1299–1308

  35. Papa JP, Rosa GH, Marana AN, Scheirer W, Cox DD (2015) Model selection for discriminative restricted Boltzmann machines through meta-heuristic techniques. J Comput Sci 9:14–18

    Article  Google Scholar 

  36. Salakhutdinov R, Larochelle H (2010) Efficient learning of deep Boltzmann machines. In: International conference on artificial intelligence and statistics, pp 693–700

  37. Salakhutdinov R, Murray I (2008) On the quantitative analysis of deep belief networks. In: Proceedings of the 25th international conference on Machine learning, pp 872–879. ACM

  38. Schmah T, Hinton GE, Zemel RS, Small SL, Strother SC (2008) Generative versus discriminative training of rbms for classification of fmri images. In: Conference on neural information processing systems, Vancouver, British Columbia, Canada, December, pp 1409–1416

  39. Smolensky P (1986) Information processing in dynamical systems: foundations of harmony theory. Technical Report, DTIC Document

  40. Specht DF (1990) Probabilistic neural networks. Neural Netw 3(1):109–118

    Article  Google Scholar 

  41. Tomczak JM, Zieba M (2015) Classification restricted Boltzmann machine for comprehensible credit scoring model. Expert Syst Appl 42(4):1789–1796

    Article  Google Scholar 

  42. Yi Z (2010) Foundations of implementing the competitive layer model by lotka-volterra recurrent neural networks. IEEE Trans Neural Netw 21(3):494–507

    Article  Google Scholar 

  43. Yi Z, Lv JC, Zhang L (2006) Output convergence analysis for a class of delayed recurrent neural networks with time-varying inputs. IEEE Trans Syst Man Cybern Part B Cybern 36(1):87–95

    Article  Google Scholar 

  44. Yu Q, Hou Y, Zhao X, Cheng G (2014) Rényi divergence based generalization for learning of classification restricted Boltzmann machines. In: 2014 IEEE International conference on data mining workshop (ICDMW), pp 692–697. IEEE

  45. Zhang GP (2000) Neural networks for classification: a survey. IEEE Trans Syst Man Cybern Part C Appl Revi 30(4):451–462

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Science Foundation of China (Grant No. 61375065) partially supported by the State Key Program of National Science Foundation of China (Grant Nos. 61432012 and 61432014).

Author information

Authors and Affiliations

  1. Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, 610065, People’s Republic of China

    Jing Yin, Jiancheng Lv, Yongsheng Sang & Jixiang Guo

Authors
  1. Jing Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiancheng Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongsheng Sang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jixiang Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiancheng Lv.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yin, J., Lv, J., Sang, Y. et al. Classification model of restricted Boltzmann machine based on reconstruction error. Neural Comput & Applic 29, 1171–1186 (2018). https://doi.org/10.1007/s00521-016-2628-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-016-2628-6

Keywords

Search

Navigation