Skip to main content
SpringerLink
Log in

New prior distribution for Bayesian neural network and learning via Hamiltonian Monte Carlo

  • Original Paper
  • Published:
Evolving Systems Aims and scope Submit manuscript

Abstract

A prior distribution of weights for Multilayer feedforward neural network in Bayesian point of view plays a central role toward generalization. In this context, we propose a new prior law for weights parameters which motivate the network regularization more than \(l_ {1}\) and \(l_ {2}\) early proposed. To train the network, we have based on Hamiltonian Monte Carlo, it is used to simulate the prior and the posterior distribution. The generated samples are used to approximate the gradient of the evidence which allows to re-estimate the hyperparameters that balance a trade off between the likelihood term and regularized term, on the other hand we use the obtained posterior samples to estimate the network output. The case problem studied in this paper includes a regression and classification tasks. The obtained results illustrate the advantages of our approach in term of error rate compared to old approach, unfortunately our method consomme time before convergence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Angelov P (2014) Outside the box: an alternative data analytics framework. J Autom Mob Robot Intell Syst 8(2):29–35

    MathSciNet  Google Scholar 

  • Angelov P, Sadeghi-Tehran P, Ramezani R (2011) An approach to automatic real-time novelty detection, object identification, and tracking in video streams based on recursive density estimation and evolving takagi-sugeno fuzzy systems. Int J Intell Syst 26(3):189–205

    Article  Google Scholar 

  • Angelov P, Kasabov N (2005) Evolving computational intelligence systems. In Proceedings of the 1st international workshop on genetic fuzzy systems, pp 76–82

  • Bache K, Lichman M (2013) UCI machine learning repository. https://archive.ics.uci.edu/ml/datasets.html. Accessed 2013

  • Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, Oxford

    MATH  Google Scholar 

  • Buntine WL, Weigend AS (1991) Bayesian back-propagation. Complex Syst 5(6):603–643

    MATH  Google Scholar 

  • de Campos Souza PV, Silva GRL, Torres LCB (2018) Uninorm based regularized fuzzy neural networks. In: 2018 IEEE conference on evolving and adaptive intelligent systems (EAIS), pp 1–8, IEEE

  • de Campos Souza PV, Torres LCB, Guimaraes AJ, Araujo VS, Araujo VJS, Rezende TS (2019) Data density-based clustering for regularized fuzzy neural networks based on nullneurons and robust activation function. Soft Comput. https://doi.org/10.1007/s00500-019-03792-z

    Article  Google Scholar 

  • Duane S, Kennedy AD, Pendleton BJ, Roweth D (1987) Hybrid Monte Carlo. Phys Lett B 195(2):216–222

    Article  MathSciNet  Google Scholar 

  • Ekonomou L, Christodoulou CA, Mladenov V (2016) An artificial neural network software tool for the assessment of the electric field around metal oxide surge arresters. Neural Comput Appl 27(5):1143–1148

    Article  Google Scholar 

  • Fan Q, Zurada JM, Wu W (2014) Convergence of online gradient method for feedforward neural networks with smoothing l1/2 regularization penalty. Neurocomputing 131:208–216

    Article  Google Scholar 

  • Kocadağlı O, Aşıkgil B (2014) Nonlinear time series forecasting with bayesian neural networks. Expert Syst Appl 41(15):6596–6610

    Article  Google Scholar 

  • Lan S (2013) Advanced Bayesian computational methods through geometric techniques. University of California, Irvine

    Google Scholar 

  • Lan S, Stathopoulos V, Shahbaba B, Girolami M (2015) Markov chain Monte Carlo from lagrangian dynamics. J Comput Graph Stat 24(2):357–378

    Article  MathSciNet  Google Scholar 

  • Leimkuhler B, Reich S (2004) Simulating hamiltonian dynamics, vol 14. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Lughofer E (2013) On-line assurance of interpretability criteria in evolving fuzzy systems-achievements, new concepts and open issues. Inf Sci 251:22–46

    Article  MathSciNet  Google Scholar 

  • MacKay DJ (1992a) The evidence framework applied to classification networks. Neural Comput 4(5):720–736

    Article  Google Scholar 

  • MacKay DJ (1992b) A practical bayesian framework for backpropagation networks. Neural Comput 4(3):448–472

    Article  Google Scholar 

  • Marwala T (2007) Bayesian training of neural networks using genetic programming. Pattern Recognit Lett 28(12):1452–1458

    Article  Google Scholar 

  • Neal RM (1993) Probabilistic inference using Markov chain Monte Carlo methods. Department of Computer Science, University of Toronto Toronto, Ontario, Canada

    Google Scholar 

  • Neal RM et al (2011) Mcmc using hamiltonian dynamics. Handb Markov Chain Monte Carlo 2(11):2

    MATH  Google Scholar 

  • Neal RM (2012) Bayesian learning for neural networks, vol 118. Springer Science & Business Media, Berlin

    Google Scholar 

  • Niu D-X, Shi H-F, Wu DD (2012) Short-term load forecasting using bayesian neural networks learned by hybrid monte carlo algorithm. Appl Soft Comput 12(6):1822–1827

    Article  Google Scholar 

  • Precup R-E, Filip H-I, Rădac M-B, Petriu EM, Preitl S, Dragoş C-A (2014) Online identification of evolving takagi-sugeno-kang fuzzy models for crane systems. Appl Soft Comput 24:1155–1163

    Article  Google Scholar 

  • Ramchoun H, Idrissi MAJ, Ghanou Y, Ettaouil M (2017) New modeling of multilayer perceptron architecture optimization with regularization: an application to pattern classification. IAENG Int J Comput Sci 44(3):261–269

    Google Scholar 

  • Ramchoun H, Ettaouil M (2018) Hamiltonian monte carlo based on evidence framework for bayesian learning to neural network. Soft Comput. https://doi.org/10.1007/s00500-018-3138-5

    Article  MATH  Google Scholar 

  • Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodological) 58:267–288

    MathSciNet  MATH  Google Scholar 

  • Vivarelli F, Williams CK (2001) Comparing bayesian neural network algorithms for classifying segmented outdoor images. Neural Netw 14(4–5):427–437

    Article  Google Scholar 

  • Wu W, Fan Q, Zurada JM, Wang J, Yang D, Liu Y (2014) Batch gradient method with smoothing l1/2 regularization for training of feedforward neural networks. Neural Netw 50:72–78

    Article  Google Scholar 

  • Zhang H, Tang Y (2017) Online gradient method with smoothing \(l_{0}\) regularization for feedforward neural networks. Neurocomputing 224:1–8

    Article  Google Scholar 

  • Zhang H, Tang Y, Liu X (2015) Batch gradient training method with smoothing \(l_{0}\) regularization for feedforward neural networks. Neural Comput Appl 26(2):383–390

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Modeling and Scientific Computing Laboratory, Faculty of Sciences and Technics, University of Sidi Mohamed Ben Abdellah, Fez, Morocco

    Hassan Ramchoun & Mohamed Ettaouil

Authors
  1. Hassan Ramchoun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Ettaouil
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hassan Ramchoun.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ramchoun, H., Ettaouil, M. New prior distribution for Bayesian neural network and learning via Hamiltonian Monte Carlo. Evolving Systems 11, 661–671 (2020). https://doi.org/10.1007/s12530-019-09288-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12530-019-09288-3

Keywords

Search

Navigation