Abstract
In order to achieve good generalization with neural networks overfitting must be controlled. Weight penalty factors are one common method of providing this control. However, using weight penalties creates the additional search problem of finding the optimal penalty factors. MacKay [5] proposed an approximate Bayesian framework for training neural networks, in which penalty factors are treated as hyperparameters and found in an iterative search. However, for classification networks trained with cross-entropy error, this search is slow and unstable, and it is not obvious how to improve it. This paper describes and compares several strategies for controlling this search. Some of these strategies greatly improve the speed and stability of the search. Test runs on a range of tasks are described
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
C. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
W. L. Buntine and A. S. Weigend. Computing second derivatives in feed-forward networks: A review. IEEE Transactions on Neural Networks, 5(3):480–488, 1994.
T. J. Hastie and R. J. Tibshirani. Generalized additive models. Chapman and Hall, London, 1990.
J. N. Hwang, S.-R. Lay, M. Maechler, R. D. Martin, and J. Schimert. Regression modeling in back-propagation and projection pursuit learning. IEEE Transactions on Neural Networks, 5(3):342–353, 1994.
D. J. C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472, 1992.
D. J. C. MacKay. Bayesian methods for backpropagation networks. In E. Domany, J. L. van Hemmen, and K. Schulten, editors, Models of Neural Networks III, chapter 6. Springer-Verlag, New York, 1994.
D. J. C. MacKay. Probable networks and plausible predictions-a review of practical Bayesian methods for supervised neural networks. Network: Computation in Neural Systems, 6:469–505, 1995.
D. J. C. MacKay. Bayesian non-linear modelling for the 1993 energy prediction competition. In G. Heidbreder, editor, Maximum Entropy and Bayesian Methods, Santa Barbara 1993, pages 221–234, Dordrecht, 1996. Kluwer.
R. M. Neal. Monte carlo implementation of gaussian process models for bayesian regression and classification. Technical Report TR9702, Dept. of Statistics, University of Toronto, 1997. Software available at http://www.cs.utoronto.ca/~radford/.
R. M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag, New York, 1996.
T. Plate, J. Bert, J. Grace, and P. Band. A comparison between neural networks and other statistical techniques for modeling the relationship between tobacco and alcohol and cancer. In M.C. Mozer, M.I. Jordan, and T. Petsche, editors, Advances in Neural Information Processing 9 (NIPS*96). MIT Press, 1997.
C. Roosen and T. Hastie. Logistic response projection pursuit. Technical report, AT&T Bell Laboratories, 1993.
H. H. Thodberg. A review of bayesian neural networks with an application to near infrared spectroscopy. IEEE Transactions on Neural Networks, 7(l):56–72, 1996.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Plate, T. (1998). Controlling the hyperparameter search in MacKay’s Bayesian neural network framework. In: Orr, G.B., Müller, KR. (eds) Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science, vol 1524. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-49430-8_5
Download citation
DOI: https://doi.org/10.1007/3-540-49430-8_5
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-65311-0
Online ISBN: 978-3-540-49430-0
eBook Packages: Springer Book Archive