Abstract
Efficient global optimization techniques such as graph cut exist for energies corresponding to binary image segmentation from low-level cues. However, introducing a high-level prior such as a shape prior or a color-distribution prior into the segmentation process typically results in an energy that is much harder to optimize. The main contribution of the paper is a new global optimization framework for a wide class of such energies. The framework is built upon two powerful techniques: graph cut and branch-and-bound. These techniques are unified through the derivation of lower bounds on the energies. Being computable via graph cut, these bounds are used to prune branches within a branch-and-bound search.
We demonstrate that the new framework can compute globally optimal segmentations for a variety of segmentation scenarios in a reasonable time on a modern CPU. These scenarios include unsupervised segmentation of an object undergoing 3D pose change, category-specific shape segmentation, and the segmentation under intensity/color priors defined by Chan-Vese and GrabCut functionals.










Similar content being viewed by others
Notes
The C++ code for this framework is available at the webpage of the first author.
In fact, the global minimum of the GrabCut functional over the set of all Gaussian mixtures is not well defined, because fitting Gaussian mixture to data without additional regularization can achieve arbitrarily high likelihood/low energy. Restricting the set of all mixtures to a large discrete subset done in our case can thus be regarded as a variant of a necessary regularization on mixture parameters.
References
Agarwal, S., Awan, A., Roth, D.: Learning to detect objects in images via a sparse, part-based representation. IEEE Trans. Pattern Anal. Mach. Intell. 26(11), 1475–1490 (2004)
Agarwal, S., Chandraker, M.K., Kahl, F., Kriegman, D.J., Belongie, S.: Practical global optimization for multiview geometry. In: ECCV (1), pp. 592–605 (2006)
Bae, E., Tai, X.C.: Efficient global minimization for the multiphase Chan-Vese model of image segmentation. In: EMMCVPR, pp. 28–41 (2009)
Boros, E., Hammer, P.L.: Pseudo-boolean optimization. Discrete Appl. Math. 123(1–3), 155–225 (2002)
Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In: ICCV, pp. 105–112 (2001)
Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: ICCV, pp. 26–33 (2003)
Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004)
Bray, M., Kohli, P., Torr, P.H.S.: Posecut: Simultaneous segmentation and 3d pose estimation of humans using dynamic graph-cuts. In: ECCV (2), pp. 642–655 (2006)
Brown, E.S., Chan, T.F., Bresson, X.: Completely convex formulation of the Chan-Vese image segmentation model. Int. J. Comput. Vis. (2011). doi:10.1007/s11263-011-0499-y
Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)
Clausen, J.: Branch and bound algorithms—principles and examples (2003)
Cremers, D., Osher, S., Soatto, S.: Kernel density estimation and intrinsic alignment for shape priors in level set segmentation. Int. J. Comput. Vis. 69(3), 335–351 (2006)
Cremers, D., Schmidt, F.R., Barthel, F.: Shape priors in variational image segmentation: Convexity, Lipschitz continuity and globally optimal solutions. In: CVPR (2008)
Darbon, J.: A note on the discrete binary mumford-shah model. In: Proceedings of Computer Vision/Computer Graphics Collaboration Techniques, (MIRAGE 2007). LNCS, vol. 44182, pp. 283–294 (2007)
Delong, A., Boykov, Y.: Globally optimal segmentation of multi-region objects. In: ICCV (2009)
El-Zehiry, N.Y., Sahoo, P., Elmaghraby, A.: Combinatorial optimization of the piecewise constant mumford-shah functional with application to scalar/vector valued and volumetric image segmentation. Image Vis. Comput. 29(6), 365–381 (2011)
Felzenszwalb, P.F.: Representation and detection of deformable shapes. IEEE Trans. Pattern Anal. Mach. Intell. 27(2), 208–220 (2005)
Felzenszwalb, P., Veksler, O.: Tiered scene labeling with dynamic programming. In: CVPR (2010)
Freedman, D.: Effective tracking through tree-search. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 604–615 (2003)
Freedman, D., Zhang, T.: Interactive graph cut based segmentation with shape priors. In: CVPR (1), pp. 755–762 (2005)
Gallo, G., Grigoriadis, M.D., Tarjan, R.E.: A fast parametric maximum flow algorithm and applications. SIAM J. Comput. 18(1), 30–55 (1989)
Gavrila, D., Philomin, V.: Real-time object detection for “smart” vehicles. In: ICCV, pp. 87–93 (1999)
Grady, L., Alvino, C.V.: The piecewise smooth mumford-shah functional on an arbitrary graph. IEEE Trans. Image Process. 18(11), 2547–2561 (2009)
Greig, D.M., Porteous, B.T., Seheult, A.H.: Exact maximum a posteriori estimation for binary images. J. R. Stat. Soc. 51(2), 271–279 (1989)
Huang, R., Pavlovic, V., Metaxas, D.N.: A graphical model framework for coupling mrfs and deformable models. In: CVPR (2), pp. 739–746 (2004)
Kim, J., Zabih, R.: A segmentation algorithm for contrast-enhanced images. In: ICCV, pp. 502–509 (2003)
Kohli, P., Torr, P.H.S.: Efficiently solving dynamic Markov random fields using graph cuts. In: ICCV, pp. 922–929 (2005)
Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts. In: ECCV (3), pp. 65–81 (2002)
Kolmogorov, V., Boykov, Y., Rother, C.: Applications of parametric maxflow in computer vision. In: ICCV, pp. 1–8 (2007)
Kumar, M.P., Torr, P.H.S., Zisserman, A.: Obj cut. In: CVPR (1), pp. 18–25 (2005)
Lampert, C.H., Blaschko, M.B., Hofmann, T.: Beyond sliding windows: Object localization by efficient subwindow search. In: CVPR (2008)
Leibe, B., Leonardis, A., Schiele, B.: Robust object detection with interleaved categorization and segmentation. Int. J. Comput. Vis. 77(1–3), 259–289 (2008)
Lempitsky, V.S., Blake, A., Rother, C.: Image segmentation by branch-and-mincut. In: ECCV (4), pp. 15–29 (2008)
Lempitsky, V., Blake, A., Rother, C.: Exact optimization for Markov random fields with non-local parameters. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press, Cambridge (2011)
Leventon, M.E., Grimson, W.E.L., Faugeras, O.D.: Statistical shape influence in geodesic active contours. In: CVPR, pp. 1316–1323 (2000)
Manning, C.D., Raghavan, P., Schutze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)
Osher, S., Sethian, J.A.: Fronts propagating with curvature dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79(1), 12–49 (1988)
Rother, C., Kolmogorov, V., Blake, A.: “grabcut”: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)
Sara Vicente, V.K., Rother, C.: Joint optimization of segmentation and appearance models. In: ICCV (2009)
Schoenemann, T., Cremers, D.: Globally optimal image segmentation with an elastic shape prior. In: ICCV, pp. 1–6 (2007)
Schoenemann, T., Schmidt, F.R., Cremers, D.: Image segmentation with elastic shape priors via global geodesics in product spaces. In: British Machine Vision Conference (BMVC), Leeds, UK (2008)
Sinop, A.K., Grady, L.: Uninitialized, globally optimal, graph-based rectilinear shape segmentation the opposing metrics method. In: ICCV, pp. 1–8 (2007)
Strandmark, P., Kahl, F., Overgaard, N.C.: Optimizing parametric total variation models. In: ICCV (2009)
Wang, Y., Staib, L.H.: Boundary finding with correspondence using statistical shape models. In: CVPR, pp. 338–345 (1998)
Zeng, X., Chen, W., Peng, Q.: Efficiently solving the piecewise constant mumford-shah model using graph cuts. Tech. Rep. (2006)
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Corollary
The bound (2) is monotonic, i.e. if Ω 1⊂Ω 2 then L(Ω 1)≥L(Ω 2).
Proof
Let us denote with A(x,Ω) the expression within the outer minimum of (2):

Then, (2) reformulates as:
Assume Ω 1⊂Ω 2. Then, for any fixed x, for all pixels p and edges \(p,q \in \mathcal{E}\), the following inequalities hold:




This is because, firstly, all values x p , 1−x p , and |x p −x q | are non-negative (recall that all x p takes the value of 0 or 1) and, secondly, all minima on the left side are taken over a subset of the domain of the same minima on the right side.
Summing up inequalities (17)–(20) over all pixels p and edges p,q and taking into account the definition (15), we get:
i.e. monotonicity holds for any fixed x.
Let x 1 be the segmentation delivering the global optimum of A(x,Ω 1): \(\mathbf{x}_{1} = \arg\min_{\mathbf{x}\in{2 ^{\mathcal{V}}}} A(\mathbf{x}, \varOmega _{1})\). Let x 2 be the segmentation delivering the global optimum of A(x,Ω 2): \(\mathbf{x}_{2} = \arg\min_{\mathbf{x}\in{2 ^{\mathcal{V}}}} A(\mathbf{x}, \varOmega _{2})\). Then, from the definition (15) and the monotonicity (21), one gets:

which concludes the proof. □
Rights and permissions
About this article
Cite this article
Lempitsky, V., Blake, A. & Rother, C. Branch-and-Mincut: Global Optimization for Image Segmentation with High-Level Priors. J Math Imaging Vis 44, 315–329 (2012). https://doi.org/10.1007/s10851-012-0328-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10851-012-0328-0