Skip to main content
SpringerLink
Log in

Branch-and-Mincut: Global Optimization for Image Segmentation with High-Level Priors

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Efficient global optimization techniques such as graph cut exist for energies corresponding to binary image segmentation from low-level cues. However, introducing a high-level prior such as a shape prior or a color-distribution prior into the segmentation process typically results in an energy that is much harder to optimize. The main contribution of the paper is a new global optimization framework for a wide class of such energies. The framework is built upon two powerful techniques: graph cut and branch-and-bound. These techniques are unified through the derivation of lower bounds on the energies. Being computable via graph cut, these bounds are used to prune branches within a branch-and-bound search.

We demonstrate that the new framework can compute globally optimal segmentations for a variety of segmentation scenarios in a reasonable time on a modern CPU. These scenarios include unsupervised segmentation of an object undergoing 3D pose change, category-specific shape segmentation, and the segmentation under intensity/color priors defined by Chan-Vese and GrabCut functionals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. The C++ code for this framework is available at the webpage of the first author.

  2. In fact, the global minimum of the GrabCut functional over the set of all Gaussian mixtures is not well defined, because fitting Gaussian mixture to data without additional regularization can achieve arbitrarily high likelihood/low energy. Restricting the set of all mixtures to a large discrete subset done in our case can thus be regarded as a variant of a necessary regularization on mixture parameters.

References

  1. Agarwal, S., Awan, A., Roth, D.: Learning to detect objects in images via a sparse, part-based representation. IEEE Trans. Pattern Anal. Mach. Intell. 26(11), 1475–1490 (2004)

    Article  Google Scholar 

  2. Agarwal, S., Chandraker, M.K., Kahl, F., Kriegman, D.J., Belongie, S.: Practical global optimization for multiview geometry. In: ECCV (1), pp. 592–605 (2006)

    Google Scholar 

  3. Bae, E., Tai, X.C.: Efficient global minimization for the multiphase Chan-Vese model of image segmentation. In: EMMCVPR, pp. 28–41 (2009)

    Google Scholar 

  4. Boros, E., Hammer, P.L.: Pseudo-boolean optimization. Discrete Appl. Math. 123(1–3), 155–225 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  5. Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In: ICCV, pp. 105–112 (2001)

    Google Scholar 

  6. Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: ICCV, pp. 26–33 (2003)

    Google Scholar 

  7. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004)

    Article  Google Scholar 

  8. Bray, M., Kohli, P., Torr, P.H.S.: Posecut: Simultaneous segmentation and 3d pose estimation of humans using dynamic graph-cuts. In: ECCV (2), pp. 642–655 (2006)

    Google Scholar 

  9. Brown, E.S., Chan, T.F., Bresson, X.: Completely convex formulation of the Chan-Vese image segmentation model. Int. J. Comput. Vis. (2011). doi:10.1007/s11263-011-0499-y

    Google Scholar 

  10. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)

    Article  MATH  Google Scholar 

  11. Clausen, J.: Branch and bound algorithms—principles and examples (2003)

  12. Cremers, D., Osher, S., Soatto, S.: Kernel density estimation and intrinsic alignment for shape priors in level set segmentation. Int. J. Comput. Vis. 69(3), 335–351 (2006)

    Article  Google Scholar 

  13. Cremers, D., Schmidt, F.R., Barthel, F.: Shape priors in variational image segmentation: Convexity, Lipschitz continuity and globally optimal solutions. In: CVPR (2008)

    Google Scholar 

  14. Darbon, J.: A note on the discrete binary mumford-shah model. In: Proceedings of Computer Vision/Computer Graphics Collaboration Techniques, (MIRAGE 2007). LNCS, vol. 44182, pp. 283–294 (2007)

    Chapter  Google Scholar 

  15. Delong, A., Boykov, Y.: Globally optimal segmentation of multi-region objects. In: ICCV (2009)

    Google Scholar 

  16. El-Zehiry, N.Y., Sahoo, P., Elmaghraby, A.: Combinatorial optimization of the piecewise constant mumford-shah functional with application to scalar/vector valued and volumetric image segmentation. Image Vis. Comput. 29(6), 365–381 (2011)

    Article  Google Scholar 

  17. Felzenszwalb, P.F.: Representation and detection of deformable shapes. IEEE Trans. Pattern Anal. Mach. Intell. 27(2), 208–220 (2005)

    Article  Google Scholar 

  18. Felzenszwalb, P., Veksler, O.: Tiered scene labeling with dynamic programming. In: CVPR (2010)

    Google Scholar 

  19. Freedman, D.: Effective tracking through tree-search. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 604–615 (2003)

    Article  Google Scholar 

  20. Freedman, D., Zhang, T.: Interactive graph cut based segmentation with shape priors. In: CVPR (1), pp. 755–762 (2005)

    Google Scholar 

  21. Gallo, G., Grigoriadis, M.D., Tarjan, R.E.: A fast parametric maximum flow algorithm and applications. SIAM J. Comput. 18(1), 30–55 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  22. Gavrila, D., Philomin, V.: Real-time object detection for “smart” vehicles. In: ICCV, pp. 87–93 (1999)

    Google Scholar 

  23. Grady, L., Alvino, C.V.: The piecewise smooth mumford-shah functional on an arbitrary graph. IEEE Trans. Image Process. 18(11), 2547–2561 (2009)

    Article  MathSciNet  Google Scholar 

  24. Greig, D.M., Porteous, B.T., Seheult, A.H.: Exact maximum a posteriori estimation for binary images. J. R. Stat. Soc. 51(2), 271–279 (1989)

    Google Scholar 

  25. Huang, R., Pavlovic, V., Metaxas, D.N.: A graphical model framework for coupling mrfs and deformable models. In: CVPR (2), pp. 739–746 (2004)

    Google Scholar 

  26. Kim, J., Zabih, R.: A segmentation algorithm for contrast-enhanced images. In: ICCV, pp. 502–509 (2003)

    Google Scholar 

  27. Kohli, P., Torr, P.H.S.: Efficiently solving dynamic Markov random fields using graph cuts. In: ICCV, pp. 922–929 (2005)

    Google Scholar 

  28. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts. In: ECCV (3), pp. 65–81 (2002)

    Google Scholar 

  29. Kolmogorov, V., Boykov, Y., Rother, C.: Applications of parametric maxflow in computer vision. In: ICCV, pp. 1–8 (2007)

    Google Scholar 

  30. Kumar, M.P., Torr, P.H.S., Zisserman, A.: Obj cut. In: CVPR (1), pp. 18–25 (2005)

    Google Scholar 

  31. Lampert, C.H., Blaschko, M.B., Hofmann, T.: Beyond sliding windows: Object localization by efficient subwindow search. In: CVPR (2008)

    Google Scholar 

  32. Leibe, B., Leonardis, A., Schiele, B.: Robust object detection with interleaved categorization and segmentation. Int. J. Comput. Vis. 77(1–3), 259–289 (2008)

    Article  Google Scholar 

  33. Lempitsky, V.S., Blake, A., Rother, C.: Image segmentation by branch-and-mincut. In: ECCV (4), pp. 15–29 (2008)

    Google Scholar 

  34. Lempitsky, V., Blake, A., Rother, C.: Exact optimization for Markov random fields with non-local parameters. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press, Cambridge (2011)

    Google Scholar 

  35. Leventon, M.E., Grimson, W.E.L., Faugeras, O.D.: Statistical shape influence in geodesic active contours. In: CVPR, pp. 1316–1323 (2000)

    Google Scholar 

  36. Manning, C.D., Raghavan, P., Schutze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)

    Book  MATH  Google Scholar 

  37. Osher, S., Sethian, J.A.: Fronts propagating with curvature dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79(1), 12–49 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  38. Rother, C., Kolmogorov, V., Blake, A.: “grabcut”: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)

    Article  Google Scholar 

  39. Sara Vicente, V.K., Rother, C.: Joint optimization of segmentation and appearance models. In: ICCV (2009)

    Google Scholar 

  40. Schoenemann, T., Cremers, D.: Globally optimal image segmentation with an elastic shape prior. In: ICCV, pp. 1–6 (2007)

    Google Scholar 

  41. Schoenemann, T., Schmidt, F.R., Cremers, D.: Image segmentation with elastic shape priors via global geodesics in product spaces. In: British Machine Vision Conference (BMVC), Leeds, UK (2008)

    Google Scholar 

  42. Sinop, A.K., Grady, L.: Uninitialized, globally optimal, graph-based rectilinear shape segmentation the opposing metrics method. In: ICCV, pp. 1–8 (2007)

    Google Scholar 

  43. Strandmark, P., Kahl, F., Overgaard, N.C.: Optimizing parametric total variation models. In: ICCV (2009)

    Google Scholar 

  44. Wang, Y., Staib, L.H.: Boundary finding with correspondence using statistical shape models. In: CVPR, pp. 338–345 (1998)

    Google Scholar 

  45. Zeng, X., Chen, W., Peng, Q.: Efficiently solving the piecewise constant mumford-shah model using graph cuts. Tech. Rep. (2006)

Download references

Author information

Authors and Affiliations

  1. Yandex, Moscow, Russia

    Victor Lempitsky

  2. Microsoft Research, Cambridge, UK

    Andrew Blake & Carsten Rother

Authors
  1. Victor Lempitsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew Blake
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carsten Rother
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Victor Lempitsky.

Appendix

Appendix

Corollary

The bound (2) is monotonic, i.e. if Ω 1⊂Ω 2 then L(Ω 1)≥L(Ω 2).

Proof

Let us denote with A(x,Ω) the expression within the outer minimum of (2):

(15)

Then, (2) reformulates as:

$$L(\varOmega ) = \min_{\mathbf{x}\in2^\mathcal{V}} A(\mathbf{x}, \varOmega ) .$$
(16)

Assume Ω 1⊂Ω 2. Then, for any fixed x, for all pixels p and edges \(p,q \in \mathcal{E}\), the following inequalities hold:

(17)
(18)
(19)
(20)

This is because, firstly, all values x p , 1−x p , and |x p −x q | are non-negative (recall that all x p takes the value of 0 or 1) and, secondly, all minima on the left side are taken over a subset of the domain of the same minima on the right side.

Summing up inequalities (17)–(20) over all pixels p and edges p,q and taking into account the definition (15), we get:

$$\forall \mathbf{x}\quad A(\mathbf{x},\varOmega _1) \ge A(\mathbf{x},\varOmega _2) ,$$
(21)

i.e. monotonicity holds for any fixed x.

Let x 1 be the segmentation delivering the global optimum of A(x,Ω 1): \(\mathbf{x}_{1} = \arg\min_{\mathbf{x}\in{2 ^{\mathcal{V}}}} A(\mathbf{x}, \varOmega _{1})\). Let x 2 be the segmentation delivering the global optimum of A(x,Ω 2): \(\mathbf{x}_{2} = \arg\min_{\mathbf{x}\in{2 ^{\mathcal{V}}}} A(\mathbf{x}, \varOmega _{2})\). Then, from the definition (15) and the monotonicity (21), one gets:

(22)

which concludes the proof. □

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lempitsky, V., Blake, A. & Rother, C. Branch-and-Mincut: Global Optimization for Image Segmentation with High-Level Priors. J Math Imaging Vis 44, 315–329 (2012). https://doi.org/10.1007/s10851-012-0328-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-012-0328-0

Keywords

Search

Navigation