Abstract
Although the Bayesian approach provides optimal performance for many inference problems, the computation cost is sometimes impractical. We herein develop a practical algorithm by which to approximate Bayesian inference in large single-layer feed-forward networks (perceptrons) based on belief propagation (BP). Although direct application of BP to the inference problem remains computationally difficult, by introducing methods and concepts from statistical mechanics that are related to the central limit theorem and the law of large numbers, the proposed BP-based algorithm exhibits nearly optimal performance in a practical time scale for ideal large networks. In order to demonstrate the practical significance of the proposed algorithm, an application to a problem that arises in a mobile communications system is also presented.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Iba, Y.: The Nishimori line and Bayesian statistics. J. Phys. A 38, 3875–3888 (1999)
Berger, J.O.: Statistical Decision Theory and Bayesian Analysis, 2nd edn. Springer, New York (1985)
Opper, M., Winther, O.: A mean field algorithm for Bayes learning in large feedforward neural networks. In: Mozer, M.C., et al. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 225–231. MIT Press, Cambridge (1997)
Opper, M., Winther, O.: A mean field approach to Bayes learning in feed-forward neural networks. Phys. Rev. Lett. 76, 1964–1967 (1996)
Opper, M., Winther, O.: Tractable approximation for probabilistic models: The adaptive TAP mean field approach. Phys. Rev. Lett. 86, 3695–3699 (2001)
Kappen, H.J., RodrÃguez, F.B.: Efficient learning in Boltzmann Machines using linear response theory. In: Kearns, M.S., et al. (eds.) Advances in Neural Information Processing Systems, vol. 11, pp. 280–286. MIT Press, Cambridge (1999)
MacKay, D.J.C., Neal, R.M.: Near Shannon limit performance of low density parity check codes. Electronics Letters 32, 1645–1646 (1996)
MacKay, D.J.C.: Good error correcting codes based on very sparse matrices. IEEE Trans. Infor. Theor. 45, 399–431 (1999)
Kabashima, Y., Saad, D.: Belief Propagation vs. TAP for decoding corrupted messages. Europhys. Lett. 44, 668–674 (1998)
Kabashima, Y.: A CDMA multiuser detection algorithm on the basis of belief propagation. J. Phys. A 36, 11111–11121 (2003)
Tanaka, T.: A Statistical-Mechanics Approach to Large-System Analysis of CDMA Multiuser Detectors. IEEE Trans. Infor. Theor. 48, 2888–2910 (2002)
Opper, M., Saad, D.: Advanced Mean Field Methods. MIT Press, Cambridge (2001)
Nishimori, H.: Statistical Physics of Spin Glasses and Information Processing. Oxform Univ. Press, New York (2001)
Pearl, J.: Probabilistic Reasoning in Intelligent Systems, 2nd edn. Morgan Kaufmann, San Francisco (1988)
Gallager, R.G.: Low Density Parity Check Codes. MIT Press, Cambridge (1963)
Tanaka, K.: Probabilistic Inference by means of Cluster Variation Method and Linear Response Theory. IEICE Trans. Inform. & Syst. E86-D, 1228–1242 (2003)
Welling, M., Teh, Y.W.: Linear Response Algorithms for Approximate Inference in Graphical Models. Neural Compt. 16, 197–221 (2004)
Murayama, T.: Statistical Mechanics of the data compression theorem. J. Phys. A 35, L95–L100 (2002)
Murayama, T., Okada, M.: One step RSB scheme for the rate distortion function. J. Phys. A 36, 11123–11130 (2003)
Vicente, R., Saad, D., Kabashima, Y.: Statistical physics of irregular low-density parity-check codes. J. Phys. A 33, 6527–6542 (2000)
Mezard, M., Parisi, G., Virasoro, M.A.: Spin Glass Theory and Beyond. World Scientific, Singapore (1987)
Watkin, T.L.H., Rau, A., Biehl, M.: The statistical mechanics of learning a rule. Rev. Mod. Phys. 65, 499–568 (1993)
Ojanpera, T., Prasad, R. (eds.): WCDMA: Towards IP Mobility and Mobile Internet, Artech House, Boston, MA (2001)
Varanasi, M.K., Aazhang, B.: Near Optimum Detection in Synchronous Code- Division Multiple-Access Systems. IEEE Trans. on Commun. 39, 725–736 (1991)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kabashima, Y., Uda, S. (2004). A BP-Based Algorithm for Performing Bayesian Inference in Large Perceptron-Type Networks. In: Ben-David, S., Case, J., Maruoka, A. (eds) Algorithmic Learning Theory. ALT 2004. Lecture Notes in Computer Science(), vol 3244. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30215-5_36
Download citation
DOI: https://doi.org/10.1007/978-3-540-30215-5_36
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23356-5
Online ISBN: 978-3-540-30215-5
eBook Packages: Springer Book Archive