Skip to main content
SpringerLink
Log in

Deep Boltzmann machine for nonlinear system modelling

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Deep Boltzmann machine (DBM) has been successfully applied in classification, regression and time series modeling. For nonlinear system modelling, DBM should also have many advantages over the other neural networks, such as input features extraction and noise tolerance. In this paper, we use DBM to model nonlinear systems by calculating the probability distributions of the input and output. Two novel weight updating algorithms are proposed to obtain these distributions. We use binary encoding and conditional probability transformation methods. The proposed methods are validated with two benchmark nonlinear systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. LeCun Y, Bengio Y, Hinton GE (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  2. Wang Z, Wang X (2018) A deep stochastic weight assignment network and its application to chess playing. J Parall Distrib Comput 117:205–211

    Article  Google Scholar 

  3. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  4. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  5. Salakhutdinov R, Hinton GE (2009) Deep boltzmann machines. In: 12th International conference on artificial intelligence and statistics (AISTATS). Clearwater Beach, Florida, USA

  6. Salakhutdinov R, Hinton GE (2012) An efficient learning procedure for deep boltzmann machines. Neural Comput 24:1967–2006

    Article  MathSciNet  MATH  Google Scholar 

  7. Larochelle H, Bengio Y (2008) Classification using discriminative restricted Boltzmann machines. In: 25th International conference on machine learning, Helsinki, Finland, pp 536–543

  8. Le Roux N, Bengio Y (2008) Representational power of restricted Boltzmann machines and deep belief networks. Neural Comput 20:1631–1649

    Article  MathSciNet  MATH  Google Scholar 

  9. Erhan D, Bengio Y, Courville A, Manzagol P-A, Vincent P (2010) Why does unsupervised pre-training help deep learning? J Mach Learn Res 11:625–660

    MathSciNet  MATH  Google Scholar 

  10. Wang X, Zhang T, Wang R (2017) Non-iterative deep learning: incorporating restricted boltzmann machine into multilayer random weight neural networks. IEEE Trans Syst Man Cybern Syst. https://doi.org/10.1109/TSMC.2017.2701419

  11. Hinton GE, Sejnowski TJ (1986) Learning and relearning in Boltzmann machines. In: Parallel distributed processing: explorations in the microstructure of cognition, vol 1: foundations, pp 282–317, MIT Press, Cambridge, MA

  12. Qiu L, Zhang L, Ren Y, Suganthan PN, Amaratunga G (2014) Ensemble deep learning for regression and time series forecasting. In: 2014 IEEE symposium on computational intelligence in ensemble learning (CIEL), pp 1–6, Orlando, FL, USA

  13. Busseti E, Osband I, Wong S (2012) Deep learning for time series modeling. Technical report, Stanford University

  14. Längkvist M, Karlsson L, Loutfi A (2014) A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recogn Lett 42:11–24

    Article  Google Scholar 

  15. Romeu P et al (2013) Time-series forecasting time-series, of indoor temperature using pre-trained deep neural networks. Artificial Neural Networks and Machine Learning-ICANN. Springer, Berlin Heidelberg, pp 451–458

  16. Ljung L (1987) System identification-theory for user. Prentice Hall, Englewood Cliffs

    MATH  Google Scholar 

  17. Hintonand GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313:504–507

    Article  MathSciNet  MATH  Google Scholar 

  18. Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. In: Advances in neural information processing systems (NIPS’06). MIT Press, Cambridge, pp 153–160

  19. de la Rosa E, Yu W (2016) Randomized algorithms for nonlinear system identification with deep learning modification. Inform Sci 364:197–212

    Article  Google Scholar 

  20. Narendra KS, Parthasarathy K (1990) Identification and control of dynamical systems using neural networks. IEEE Trans Neural Netw 1(1):4-2

    Article  Google Scholar 

  21. Jagannathan S, Lewis FL (1996) Identification of nonlinear dynamical systems using multilayered neural networks. Automatica 32(12):1707–1712

    Article  MathSciNet  MATH  Google Scholar 

  22. Busseti E, Osband I, Wong S (2012) Deep learning for time series modeling. CS 229 Technical Report, Stanford University

  23. de la Rosa E, Yu W (2015) Restricted Boltzmann machine for nonlinear system modeling. In: 14th IEEE international conference on machine learning and applications (ICMLA15), Miami, USA

  24. de la Rosa E, Yu W, Li X (2016) Nonlinear system modeling with deep neural networks and autoencoders algorithm. In: 2016 IEEE international conference on systems, man, and cybernetics (SMC16), Budapest, Hungary, pp 2157–2162

  25. de la Rosa E, Yu W (2015) Nonlinear system identification using deep learning and randomized algorithms. In: 2015 IEEE international conference on information and automation, Lijing, China, pp 274–279

  26. Chen S, Billings SA (1992) Neural networks for nonlinear system modelling and identification. Int J Control 56(2):319–346

    Article  MathSciNet  MATH  Google Scholar 

  27. Cybenko G (1989) Approximation by superposition of sigmoidal activation function. Math Control Sig Syst 2:303–314

    Article  MATH  Google Scholar 

  28. Huang GB, Chen L, Siew CK (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw. 17(4):879–92

    Article  Google Scholar 

  29. Ackley DH, Hinton GE, Sejnowski TJ (1985) A learning algorithm for boltzmann machines. Cogn Sci 9:147–169

    Article  Google Scholar 

  30. Bengio Y, Delalleau O (2009) Justifying and generalizing contrastive divergence. Neural Comput 21(6):1601–1621

    Article  MathSciNet  MATH  Google Scholar 

  31. Box G, Jenkins G, Reinsel G (2008) Time series analysis: forecasting and control, 4th edn. Wiley, New York

    Book  MATH  Google Scholar 

  32. Bergstra J, Bengio Y (2011) Random search for hyper-parameter optimization. J Machine Learn Res 13:281–305

    MathSciNet  MATH  Google Scholar 

  33. Schoukens J, Suykens J, Ljung L (2009) Wiener-Hammerstein benchmark, 15th IFAC symposiumon system identification. Saint-Malo, France

    Google Scholar 

  34. Erhan D, Manzagol PA, Bengio Y, Bengio S, Vincent P (2009) The difficulty of training deep architectures and the effect of unsupervised pretraining. In: 12th International conference on artificial intelligence and statistics (AISTATS’09), pp 153–160

  35. Bartlett PL (1997) For valid generalization, the size of the weights is more important than the size of the network. In: Mozer M, Jordan M, Petsche T (eds) Advances in neural information processing systems’ 1996, vol 9. MIT Press, Cambridge, pp 134–140

    Google Scholar 

  36. Jang JS (1993) ANFIS: Adaptive-network-based fuzzy inference system. IEEE Trans Syst Man Cybern 23:665–685

    Article  Google Scholar 

  37. Mitra S, Hayashi Y (2000) Neuro-fuzzy rule generation: survey in soft computing framework. IEEE Trans Neural Netw 11(3):748–769

    Article  Google Scholar 

  38. Rivals I, Personnaz L (2003) Neural-network construction and selection in nonlinear modeling. IEEE Trans Neural Netw 14(4):804–820

    Article  MATH  Google Scholar 

  39. Wang X, Cao W (2018) Non-iterative approaches in training feed-forward neural networks and their applications. Soft Comput. https://doi.org/10.1007/s00500-018-3203-0

  40. Leung FHF, Lam HK, Ling SH, Tam PKS (2003) Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Transact Neural Netw 14:79–88

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Control Automatico, CINVESTAV-IPN (National Polytechnic Institute), Av. IPN 2508, Mexico City, 07360, Mexico

    Wen Yu & Erick de la Rosa

Authors
  1. Wen Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erick de la Rosa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen Yu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, W., de la Rosa, E. Deep Boltzmann machine for nonlinear system modelling. Int. J. Mach. Learn. & Cyber. 10, 1705–1716 (2019). https://doi.org/10.1007/s13042-018-0847-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-018-0847-0

Keywords

Search

Navigation