Invited articleBayesian approach for neural networks—review and case studies
Introduction
In Bayesian data analysis all uncertain quantities are modeled as probability distributions, and inference is performed by constructing the posterior conditional probabilities for the unobserved variables of interest, given the observed data sample and prior assumptions. Good references for Bayesian data analysis are Berger, 1985, Bernardo and Smith, 1994, Gelman et al., 1995.
For neural networks, the Bayesian approach was pioneered in Buntine and Weigend, 1991, MacKay, 1992, Neal, 1992, and reviewed in Bishop, 1995, MacKay, 1995, Neal, 1996. With neural networks, the main difficulty in model building is controlling the complexity of the model. It is well known that the optimal number of degrees of freedom in the model depends on the number of training samples, amount of noise in the samples and the complexity of the underlying function being estimated. With standard neural networks techniques, the means for both determining the correct model complexity and setting up a network with the desired complexity are rather crude and often computationally very expensive.
In the Bayesian approach, these issues can be handled in a natural and consistent way. The unknown degree of complexity is handled by defining vague (non-informative) priors for the hyperparameters that determine the model complexity, and the resulting model is averaged over all model complexities weighted by their posterior probability given the data sample. The model can be allowed to have different complexity in different parts of the model by grouping the parameters that are exchangeable (have identical role in the model) to have a common hyperparameter. If, in addition, it is assumed that the complexities are more probably similar, a hierarchical hyperprior can be defined for the variance of the hyperparameters between groups.
Another problem of standard neural network methods is the lack of tools for analyzing the results (confidence intervals for the results, like 10 and 90% quantiles, etc.). The Bayesian analysis yields posterior predictive distributions for any variables of interest, making the computation of confidence intervals possible.
In this contribution, we discuss the Bayesian approach in statistical modeling (Section 2), with emphasis on the role of prior knowledge in the modeling process. In Section 3 we give a short review of Bayesian MLP models and MCMC techniques for marginalization. Then we present three real world modeling problems, where we assess the performance of the Bayesian MLP models and compare the performance to standard neural networks methods and other statistical models. The application problems are: (i) a regression problem of predicting the quality of concrete in concrete manufacturing process (Section 4); (ii) approximating an inverse mapping in a tomographic image reconstruction problem (Section 5); and (iii) a classification problem of recognizing tree trunks in forest scenes (Section 6). Finally, we discuss the conclusions of our experiments in relation to other related studies on Bayesian neural networks.
Section snippets
The Bayesian approach
The key principle of Bayesian approach is to construct the posterior probability distributions for all the unknown entities in a model, given the data sample. To use the model, marginal distributions are constructed for all those entities that we are interested in, i.e. the end variables of the study. These can be the parameters in parametric models, or the predictions in (non-parametric) regression or classification tasks.
Use of the posterior probabilities requires explicit definition of the
Bayesian learning for MLP networks
In the following, we give a short overview of the Bayesian approach for neural networks. We concentrate on MLP networks and Markov chain Monte Carlo methods for computing the integrations, following the approach introduced in Neal (1992). A detailed treatment can be found in Neal (1996), which also describes the use of the flexible Bayesian modeling (FBM) software package,1 that was the main tool used in the case problems reviewed in this
Case I: regression task in concrete quality assumption
In this section, we report results of using Bayesian MLPs for regression in a concrete quality estimation problem. The goal of the project was to develop a model for predicting the quality properties of concrete, as a part of a large quality control program of the industrial partner of the project. The quality variables included, e.g. compression strengths and densities for 1, 28 and 91 days after casting, and bleeding (water extraction), spread, slump and air-%, that measure the properties of
Case II: inverse problem in electrical impedance tomography
In this section we report results on using Bayesian MLPs for solving the ill-posed inverse problem in electrical impedance tomography (EIT). The full report of the proposed approach is presented in Lampinen, Vehtari and Leinonen (1999).
The aim in EIT is to recover the internal structure of an object from surface measurements. A number of electrodes are attached to the surface of the object and current patterns are injected through the electrodes and the resulting potentials are measured. The
Case III: classification task in forest scene analysis
In this section, we report results of using the Bayesian MLP for classification of forest scenes. The objective of the project was to assess the accuracy of estimating the volumes of growing trees from digital images. To locate the tree trunks and to initialize the fitting of the trunk contour model, a classification of the image pixels to tree and non-tree classes was necessary. The main problem in the task was the large variance in the classes. The appearance of the tree trunks varies in
Discussion and conclusions
We have reviewed the Bayesian approach for neural networks, concentrating on the MLP model and MCMC approximation for computing the marginal distribution of the end variables of the study from the joint posterior of all unknown variables given the data. In three real applications, we have assessed the performance of the Bayesian approach and compared it to other methods.
The most important advantage of the Bayesian approach in the case studies was the possibility to handle the situation where
Acknowledgements
This study was partly funded by TEKES Grant 40888/97 (Project PROMISE, Applications of Probabilistic Modeling and Search) and Graduate School in Electronics, Telecommunications and Automation (GETA). The authors would like to thank H. Järvenpää for providing her expertise into the case study I, and K. Leinonen and J. Kaipio for aiding in the problem setup and providing the TV inverse method in the case study II.
References (57)
- et al.
Hybrid Monte Carlo
Physics Letters B
(1987) - et al.
Bayesian neural networks for classification: how useful is the evidence framework?
Neural Networks
(1999) - et al.
Ensemble learning in Bayesian neural networks
Statistical design theory and Bayesian analysis, Springer series in statistics
(1985)- et al.
On the development of reference priors
- et al.
Bayesian theory
(1994) Curvature-driven smoothing: a learning algorithm for feed-forward networks
IEEE Transactions on Neural Networks
(1993)Neural networks for pattern recognition
(1995)- et al.
Regression with input-dependent noise: a Bayesian treatment
- et al.
Classification and regression trees
(1984)