Abstract
We propose a family of convergent double-loop algorithms which minimize the TRW free energy. These algorithms are based on the concave convex procedure (CCCP) so we call them TRW-CCCP. Our formulation includes many free parameters which specify an infinite number of decompositions of the TRW free energy into convex and concave parts. TRW-CCCP is guaranteed to converge to the global minima for any settings of these free parameters, including adaptive settings if they satisfy conditions defined in this paper. We show that the values of these free parameters control the speed of convergence of the the inner and outer loops in TRW-CCCP. We performed experiments on a two-dimensional Ising model observing that TRW-CCCP converges to the global minimum of the TRW free energy and that the convergence rate depends on the parameter settings. We compare with the original message passing algorithm (TRW-BP) by varying the difficulty of the problem (by adjusting the energy function) and the number of iterations in the inner loop of TRW-CCCP. We show that on difficult problems TRW-CCCP converges faster than TRW-BP (in terms of total number of iterations) if few inner loop iterations are used.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Freeman W., Pasztor E., Carmichael O.: “Learning Low-Level Vision”. International Journal of Computer Vision 40(1), 25–47 (2000)
Tappen, M. and Freeman, W., “Comparison of Graph Cuts with Belief Propagation for Stereo, Using Identical MRF Parameters,” in Proc. of the 9th International Conference on Computer Vision (ICCV2003), pp. 900–907, 2003.
Yanover C., Meltzer T., Weiss Y.: “Linear programming relaxations and belief propagation -an empirical study”. Journal of Machine Learning Research 7, 1887–1907 (2006)
Lauritzen S., Sheehan N: “Graphical Models for Genetic Analyses”. Statistical Science 18(4), 489–514 (2003)
Steimer A., Maass W., Douglas R.: “Belief propagation in networks of spiking neurons”. Neural Comput 21(9), 2502–2523 (2009)
Yuille A.: “CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation”. Neural Comput 14(7), 1691–1722 (2002)
Heskes T.: “Convexity Arguments for Efficient Minimization of the Bethe and Kikuchi Free Energies”. Journal of Artificial Intelligence Research 26, 153–190 (2006)
Wainwright M., Jaakkola T., Willsky A.: “A New Class of Upper Bounds on the Log Partition Function”. IEEE trans. Information Theory 51(7), 2313–2335 (2005)
Yu, C. and Joachims, T. “Learning structural svms with latent variables,” in Proc. of the 26th International Conference on Machine Learning (ICML2009), pp. 1169–1176, 2009.
Nishiyama, Y. and Watanabe, S., “Generalization of Concave and Convex Decomposition in Kikuchi Free Energy” in Proc. of International Conference on Artifical Neural Networks (ICANN 2008), pp. 51–60, 2008.
Wainwright, M. and Jordan, M., “Graphical models, exponential families, and variational inference,” Technical Report, UC Berkeley, Dept. of Statistics, 2003.
Yuille A., Rangarajan A.: “The Concave-Convex Procedure” Neural Comput 15(4), 915–936 (2003)
Globerson, A. and Jaakkola, T., “Convergent propagation algorithms via oriented trees,” in Proc. of The 23rd Conference on Uncertainty in Artificial Intelligence (UAI2007), 2007.
Meltzer, T., Globerson, A. and Weiss, Y., “Convergent message passing algorithms -a unifying view,” in Proc. of The 25rd Conference on Uncertainty in Artificial Intelligence (UAI2009), 2009.
Author information
Authors and Affiliations
Corresponding author
About this article
Cite this article
Nishiyama, Y., Ye, X. & Yuille, A.L. A Family of CCCP Algorithms Which Minimize the TRW Free Energy. New Gener. Comput. 30, 3–16 (2012). https://doi.org/10.1007/s00354-012-0102-2
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00354-012-0102-2