Abstract
Bayesian neural networks learn a posterior probability distribution over the weights of the network to estimate the uncertainty in predictions. Parameterization of prior and posterior distribution as Gaussian in Monte Carlo Dropout, Bayes-by-Backprop (BBB) often fails in latent hyperspherical structure [1, 15]. In this paper, we address an enhanced approach for selecting weights of a neural network [2] corresponding to each layer with a uniform distribution on the Hypersphere to efficiently approximate the posterior distribution, called Hypersphere Bayes by Backprop. We show that this Hyperspherical Weight Uncertainty in Neural Networks is able to model a richer variational distribution than previous methods and obtain well-calibrated predictive uncertainty in deep learning in non-linear regression, image classification and high dimensional active learning. We then demonstrate how this uncertainty in the weights can be used to improve generalisation in Variational Auto-Encoder (VAE) problem.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Arthur, M.K.: Point picking and distributing on the disc and sphere. Tech. rep, Army Research Laboratory (2015)
Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural networks. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 1613–1622 (2015)
Davidson, T.R., Falorsi, L., De Cao, N., Kipf, T., Tomczak, J.M.: Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891 (2018)
Farquhar, S., Osborne, M., Gal, Y.: Radial bayesian neural networks: robust variational inference in big models. arXiv preprint arXiv:1907.00865 (2019)
Gal, Y.: Uncertainty in deep learning. Ph.D. thesis, University of Cambridge (2016)
Ghoshal, B., Tucker, A., Sanghera, B., Lup Wong, W.: Estimating uncertainty in deep learning for reporting confidence to clinicians in medical image segmentation and diseases detection. Comput. Intell. (2019)
Graves, A.: Practical variational inference for neural networks. In: Advances in Neural Information Processing Systems, pp. 2348–2356 (2011)
Hinton, G.E., Van Camp, D.: Keeping the neural networks simple by minimizing the description length of the weights. In: Proceedings of the Sixth Annual Conference on Computational Learning Theory, pp. 5–13 (1993)
Hron, J., Matthews, A.G.D.G., Ghahramani, Z.: Variational bayesian dropout: pitfalls and fixes. arXiv preprint arXiv:1807.01969 (2018)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
MacKay, D.J.: A practical bayesian framework for backpropagation networks. Neural Comput. 4(3), 448–472 (1992)
Neal, R.M.: Bayesian learning via stochastic dynamics. In: Advances in Neural Information Processing Systems, pp. 475–482 (1993)
Nitarshan, R.: Weight uncertainty in neural networks. https://www.nitarshan.com/bayes-by-backprop/ (2018)
Oh, C., Gavves, E., Welling, M.: Bock: Bayesian optimization with cylindrical kernels. arXiv preprint arXiv:1806.01619 (2018)
Weisstein, E.W.: Hypersphere. https://mathworld.wolfram.com/ (2002)
Welling, M., Teh, Y.W.: Bayesian learning via stochastic gradient langevin dynamics. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ghoshal, B., Tucker, A. (2021). Hyperspherical Weight Uncertainty in Neural Networks. In: Abreu, P.H., Rodrigues, P.P., Fernández, A., Gama, J. (eds) Advances in Intelligent Data Analysis XIX. IDA 2021. Lecture Notes in Computer Science(), vol 12695. Springer, Cham. https://doi.org/10.1007/978-3-030-74251-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-74251-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74250-8
Online ISBN: 978-3-030-74251-5
eBook Packages: Computer ScienceComputer Science (R0)