Synergistic arc-weight estimation for interactive image segmentation using graphs

https://doi.org/10.1016/j.cviu.2009.08.001Get rights and content

Abstract

We introduce a framework for synergistic arc-weight estimation, where the user draws markers inside each object (including background), arc weights are estimated from image attributes and object information (pixels under the markers), and a visual feedback guides the user’s next action. We demonstrate the method in several graph-based segmentation approaches as a basic step (which should be followed by some proper approach-specific adaptive procedure) and show its advantage over methods that do not exploit object information and over methods that recompute weights during delineation, which make the user to lose control over the segmentation process. We also validate the method using medical data from two imaging modalities (CT and MRI-T1).

Introduction

In image processing and computer vision, there are several situations in which user interaction becomes essential in obtaining effective image segmentation. The high-level, application-domain-specific knowledge of the user is often required in medical image analysis [16], [8], [29], [43] because of poorly defined structures, and in the digital matting of natural scenes [2], [55], because of their heterogeneous nature.

Image segmentation involves two tightly coupled tasks: recognition and delineation [16]. Recognition is the task of determining the approximate whereabouts of a desired object in the image, while delineation completes segmentation by precisely defining its spatial extent. Humans usually outperform computers in recognition, but the contrary can be observed in object delineation. While the user can reduce recognition to a simple click of the mouse inside the object, repeatable human delineation is challenging due to human subjectivity. On the other hand, computers can be very precise, even when they are not accurate, but often the absence of high-level object information (location, shape, appearance) makes object recognition a difficult task for computers. In order to overcome some of these shortcomings from both sides, some approaches have combined recognition by the user with delineation by the computer in a synergistic way [22], [15], [7], [45]. This topic forms the central focus of this paper.

In operator-assisted synergistic segmentation, the user usually adds/removes markers (seed pixels, anchor boundary points) for recognition, while subsequent delineation is performed by the computer in interactive time. Accuracy becomes a compromise between the user’s patience for verification and correction, and the quality of delineation. The methods usually make direct/indirect use of some image-graph concept, such as arc weight between pixels. The weight may represent different attribute functionals such as similarity, speed function, affinity, cost, distance, etc; depending on different frameworks used, such as watershed, level sets, fuzzy connectedness, graph cuts, etc. The accurate delineation by these methods with minimum user intervention strongly depends on a suitable arc-weight estimation, which usually takes into account image attributes and/or object information often obtained from markers selected by the user during segmentation [7], [45]. Object information is very crucial for improving the quality of arc-weight estimation. However, the user’s actions need guidance from visual feedback about the quality of the arc weights. Further, the markers used for delineation should never be used to recompute weights. Very often, these markers need to be selected in regions where object and background have similar properties. Weight recomputation followed by delineation based on these markers may destroy other parts of the image where the user was already satisfied with the segmentation results, making the user to lose control over the process.

We propose a synergistic approach for arc-weight estimation, which is separated from the process of interactive image segmentation itself. As a training step, the user selects markers inside each object, where image background is also considered as an object, guided by a visual feedback about the quality of arc-weight estimation. The training markers may be used to start object delineation, but markers selected during segmentation are never allowed to modify arc-weight assignment. We use this approach as a basic step in several image segmentation methods, such as those based on the min-cut/max-flow algorithm [7] and approaches which can be easily implemented by the image foresting transform (IFT) [21]. Note that our aim is not to compare segmentation methods, but to show that several of them can benefit from a disciplined, systematic, objective, and effective procedure for arc-weight assignment, followed by some proper approach-specific adaptive procedure, such as the complement of the weights, owing to the nature of the meaning of weights in some methods; or by some tuning procedure (e.g., non-maximal suppression [39], increasing transformations [39], gradient orientation (Section 4.7)). The visualization of the arc weights also allows the user to choose the most appropriate method for a given image. For example, it is desirable in live wire that the arc weights be lower along the object’s boundary than in the neighborhood around it [16], [22]; the local affinities in relative-fuzzy connectedness [47] be higher inside and outside the object than on its boundary; the gradient values in watershed transforms be higher for pixels on the object’s boundary than in its interior and exterior [32], [4], [15]; the gradient values in tree pruning be higher on the object’s boundary than in its interior, and, at least, in a neighborhood in its exterior [3]; and the arc weights in graph-cut segmentation be lower across the object’s boundary than in its interior and exterior [7], [51], [56]. Additionally, energy minimization in [7] using the min-cut/max-flow algorithm from source to sink nodes also requires higher arc weights between source and object pixels, lower arc weights between source and background pixels, lower arc weights between sink and object pixels, and higher arc weights between sink and background pixels. Clearly, the effectiveness of these approaches suffers when the above desirable conditions are not satisfied, and this explains why the visual feedback helps the user to choose the most appropriate method for a given segmentation task.

To outline this paper: Section 2 presents the basic concepts on image graphs and the terminology adopted in this paper. Arc-weight estimation is presented in Section 3 by showing how to exploit image attributes and object information provided by user-selected markers. Section 4 describes several interactive segmentation methods based on the arc-weight assignment of Section 3, and the main advantages of the synergistic approach are demonstrated in Section 5, including evaluation experiments with medical data from two imaging modalities (CT and MRI). Our conclusions are stated in Section 6.

Section snippets

Basic concepts on image graphs

An image Iˆ is a pair (DIˆ,I) where DIˆZn is the image domain and I(s) assigns a set of m scalars Ii(s),i=1,2,,m, to each pixel sDIˆ. This definition applies to multi-dimensional and multi-parametric images. For example, {I1(s),I2(s),I3(s)} may be the red, green and blue values of s in a color image Iˆ. The subindex i is dropped for gray images since it becomes awkward when m=1.

An irreflexive adjacency relation A is a binary relation between distinct pixels. We use tA(s) or (s,t)A to

Synergistic arc-weight estimation

Arc-weight estimation takes into account image attributes and object information in order to enhance the discontinuities between object and background. Let v be an algorithm which extracts attributes (color, gradient, texture) from any pixel sDIˆ and returns a vector v(s). In the simplest case, we may take v(s)=I(s). However, the best set of attributes depends on each given application.

Interactive segmentation methods

A segmentation result is represented by a label image Lˆ=(DIˆ,L), in which each label 1L(s)c assigns a pixel sDIˆ to one object out of c objects, including background. For the sake of simplicity, we have considered the case of c=2 in all examples of this paper. All methods presented in this section have been well published, so we will present only a short description with their graph parameters customized as a function of w(s,t) and μ(l|v(s)), although for other methods, different adaptive

Experiments and results

The examples in the previous sections have shown that the proposed process for arc-weight assignment is useful in several image segmentation methods. The synergism between the user and the computer offers some important advantages as well. Object information is incorporated into arc-weight estimation under user supervision and control. In traditional segmentation methods [28], [4], [51], [56], arc-weight estimation is usually treated as a simple embedded process, disregarding, in many cases,

Conclusion

We have presented an interactive method for arc-weight estimation, which can be employed effectively by several graph-based segmentation approaches as we demonstrated. Our method exploits in a synergistic way the human abilities for recognition and the computer abilities for delineation. While the user draws markers inside each object (including background), arc weights are estimated from image attributes and object information (pixels under the markers), and a visual feedback guides the user’s

Acknowledgments

The authors thank FAPESP and CNPq for the financial support. The images of Fig. 3, Fig. 4, Fig. 6, Fig. 10, Fig. 14 have been obtained from http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping.

References (59)

  • R.S. Torres et al.

    Contour salience descriptors for effective image retrieval and analysis

    Image and Vision Computing

    (2007)
  • J.K. Udupa et al.

    A framework for evaluating image segmentation algorithms

    Computerized Medical Imaging and Graphics

    (2006)
  • R. Audigier, R.A. Lotufo, Seed-relative segmentation robustness of watershed and fuzzy connectedness approaches, in: XX...
  • Xue Bai, Guillermo Sapiro, Distance cut: interactive segmentation and matting of images and videos, in: IEEE...
  • F.P.G. Bergo et al.

    Automatic image segmentation by tree pruning

    Journal of Mathematical Imaging and Vision

    (2007)
  • S. Beucher et al.

    The morphological approach to segmentation: the watershed transformation

  • Y. Boykov et al.

    An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2004)
  • Y. Boykov et al.

    Fast approximate energy minimization via graph cuts

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2001)
  • Y.Y. Boykov, M.-P. Jolly, Interactive graph cuts for optimal boundary and region segmentation of objects in N–D images,...
  • D.L. Collins et al.

    Design and construction of a realistic digital brain phantom

    IEEE Transactions on Medical Imaging

    (1998)
  • T. Cormen et al.

    Introduction to Algorithms

    (1990)
  • I.J. Cox, S.B. Rao, Y. Zhong, Ratio regions: a technique for image segmentation, in: International Conference on...
  • L.R. Dice

    Measures of the amount of ecologic association between species

    Ecology

    (1945)
  • R.O. Duda et al.

    Pattern Classification

    (2001)
  • A.X. Falcão et al.

    Interactive volume segmentation with differential image foresting transforms

    IEEE Transactions on Medical Imaging

    (2004)
  • A.X. Falcão, F.P.G. Bergo, P.A.V. Miranda, Image segmentation by tree pruning, in: Proceedings of the XVII Brazillian...
  • A.X. Falcão, B.S. da Cunha, R.A. Lotufo, Design of connected operators using the image foresting transform, in:...
  • A.X. Falcão, P.A.V. Miranda, A. Rocha, A linear-time approach for image segmentation using graph-cut measures, in:...
  • A.X. Falcão et al.

    The image foresting transform: theory, algorithms, and applications

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2004)
  • Cited by (54)

    • Grabber: A tool to improve convergence in interactive image segmentation

      2020, Pattern Recognition Letters
      Citation Excerpt :

      Initially, the methods did not learn the parameters of the segmentation model [2,3]. Many methods then started learning a model but fixing it during segmentation [1,4–7]. Other approaches can learn and update a model from user inputs during segmentation [8–12] and several recent ones start from a pre-trained model that is adapted from user inputs during segmentation [13–15].

    • Unsupervised hierarchical image segmentation through fuzzy entropy maximization

      2017, Pattern Recognition
      Citation Excerpt :

      The third group is to combine threshold-based segmentation results with graph cuts (GC). By setting additional smoothness term, the spatial coherence can be enforced [29,39,40]. It is difficult to apply fuzzy c-partition entropy to color image segmentation directly.

    View all citing articles on Scopus
    View full text