Abstract
PatchMatch (PM) is a simple, yet very powerful and successful method for optimizing continuous labelling problems. The algorithm has two main ingredients: the update of the solution space by sampling and the use of the spatial neighbourhood to propagate samples. We show how these ingredients are related to steps in a specific form of belief propagation (BP) in the continuous space, called max-product particle BP (MP-PBP). However, MP-PBP has thus far been too slow to allow complex state spaces. In the case where all nodes share a common state space and the smoothness prior favours equal values, we show that unifying the two approaches yields a new algorithm, PMBP, which is more accurate than PM and orders of magnitude faster than MP-PBP. To illustrate the benefits of our PMBP method we have built a new stereo matching algorithm with unary terms which are borrowed from the recent PM Stereo work and novel realistic pairwise terms that provide smoothness. We have experimentally verified that our method is an improvement over state-of-the-art techniques at sub-pixel accuracy level.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Note that “flow field” is intentionally left imprecise here. The key is that the globally optimum NNF is not smooth, but the approximate NNF found by PM tends to be, due to the smoothness of the underlying real-world physical process which generates the image correspondences.
This energy-based formulation can be converted to a probabilistic form using the conversions: belief \(b_s({\mathbf{u}}_s):=\exp (-B_s({\mathbf{u}}_s))\) and message \(m_{t \rightarrow s}({\mathbf{u}}_s)=\exp (-M_{t \rightarrow s}({\mathbf{u}}_s)).\)
References
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009). PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (Proceedings of SIGGRAPH), 28(3), 24.
Barnes, C., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2010). The generalized PatchMatch correspondence algorithm. In Proceedings of ECCV.
Bleyer, M., Rhemann, C., & Rother, C. (2011). PatchMatch Stereo—Stereo matching with slanted support windows. In Proceedings of BMVC.
Boltz, S., & Nielsen, F. (2010). Randomized motion estimation. In Proceedings of ICIP (pp. 781–784).
HaCohen, Y., Shechtman, E., Goldman, D. B., & Lischinski, D. (2011). Non-rigid dense correspondence with applications for image enhancement. ACM Transactions on Graphics (Proceedings of SIGGRAPH), 30(4), 70:1–70:9.
He, K., Rhemann, C., Rother, C., Tang, X., & Sun, J. (2011). A global sampling method for alpha matting. In Proceedings of CVPR (pp. 2049–2056).
Ihler, A., & McAllester, D. (2009). Particle belief propagation. In Proceedings of AISTATS (Vol. 5, pp. 256–263).
Isard, M., MacCormick, J., & Achan, K. (2008). Continuously-adaptive discretization for message-passing algorithms. In Proceedings of NIPS (pp. 737–744).
Kolmogorov, V. (2006). Convergent tree-reweighted message passing for energy minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(10), 1568–1583.
Korman, S., & Avidan, S. (2011). Coherency sensitive hashing. In Proceedings of ICCV (pp. 1607–1614).
Kothapa, R., Pachecho, J., & Sudderth, E. B. (2011). Max-product particle belief propagation. Master’s Thesis, Brown University.
Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., & Van Gool, L. (2011). Transforming image completion. In Proceedings of BMVC.
Noorshams, N., & Wainwright, M. J. (2011). Stochastic belief propagation: Low-complexity message-passing with guarantees. In Communication, Control, and Computing (Allerton), 2011 49th Annual Allerton Conference. IEEE (pp. 269–276).
Nowozin, S., & Lampert, C. (2011). Structured learning and prediction in computer vision (Vol. 6). Boston: Now publishers Inc.
Pal, C., Sutton, C., & McCallum, A. (2006). Sparse forward–backward using minimum divergence beams for fast training of conditional random fields. In Proceedings of ICASSP (Vol. 5).
Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Francisco: Morgan Kaufmann Publishers Inc.
Peng, J., Hazan, T., McAllester, D. A., & Urtasun, R. (2011). Convex max-product algorithms for continuous MRFs with applications to protein folding. In Proceedings of ICML.
Sudderth, E. B., Ihler, A. T., Isard, M., Freeman, W. T., & Willsky, A. S. (2010). Nonparametric belief propagation. Communications of the ACM, 53(10), 95–103.
Yamaguchi, K., Hazan, T., McAllester, D., & Urtasun, R. (2012). Continuous Markov random fields for robust stereo estimation. In Computer Vision—ECCV 2012 (pp. 45–58). Berlin, Heidelberg: Springer.
Yedidia, J., Freeman, W., & Weiss, Y. (2005). Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory, 51(7), 2282–2312.
Acknowledgments
We thank Christoph Rhemann and Michael Bleyer for their help with the PatchMatch Stereo code and also for fruitful discussions.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Besse, F., Rother, C., Fitzgibbon, A. et al. PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation. Int J Comput Vis 110, 2–13 (2014). https://doi.org/10.1007/s11263-013-0653-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-013-0653-9