Synergistic arc-weight estimation for interactive image segmentation using graphs
Introduction
In image processing and computer vision, there are several situations in which user interaction becomes essential in obtaining effective image segmentation. The high-level, application-domain-specific knowledge of the user is often required in medical image analysis [16], [8], [29], [43] because of poorly defined structures, and in the digital matting of natural scenes [2], [55], because of their heterogeneous nature.
Image segmentation involves two tightly coupled tasks: recognition and delineation [16]. Recognition is the task of determining the approximate whereabouts of a desired object in the image, while delineation completes segmentation by precisely defining its spatial extent. Humans usually outperform computers in recognition, but the contrary can be observed in object delineation. While the user can reduce recognition to a simple click of the mouse inside the object, repeatable human delineation is challenging due to human subjectivity. On the other hand, computers can be very precise, even when they are not accurate, but often the absence of high-level object information (location, shape, appearance) makes object recognition a difficult task for computers. In order to overcome some of these shortcomings from both sides, some approaches have combined recognition by the user with delineation by the computer in a synergistic way [22], [15], [7], [45]. This topic forms the central focus of this paper.
In operator-assisted synergistic segmentation, the user usually adds/removes markers (seed pixels, anchor boundary points) for recognition, while subsequent delineation is performed by the computer in interactive time. Accuracy becomes a compromise between the user’s patience for verification and correction, and the quality of delineation. The methods usually make direct/indirect use of some image-graph concept, such as arc weight between pixels. The weight may represent different attribute functionals such as similarity, speed function, affinity, cost, distance, etc; depending on different frameworks used, such as watershed, level sets, fuzzy connectedness, graph cuts, etc. The accurate delineation by these methods with minimum user intervention strongly depends on a suitable arc-weight estimation, which usually takes into account image attributes and/or object information often obtained from markers selected by the user during segmentation [7], [45]. Object information is very crucial for improving the quality of arc-weight estimation. However, the user’s actions need guidance from visual feedback about the quality of the arc weights. Further, the markers used for delineation should never be used to recompute weights. Very often, these markers need to be selected in regions where object and background have similar properties. Weight recomputation followed by delineation based on these markers may destroy other parts of the image where the user was already satisfied with the segmentation results, making the user to lose control over the process.
We propose a synergistic approach for arc-weight estimation, which is separated from the process of interactive image segmentation itself. As a training step, the user selects markers inside each object, where image background is also considered as an object, guided by a visual feedback about the quality of arc-weight estimation. The training markers may be used to start object delineation, but markers selected during segmentation are never allowed to modify arc-weight assignment. We use this approach as a basic step in several image segmentation methods, such as those based on the min-cut/max-flow algorithm [7] and approaches which can be easily implemented by the image foresting transform (IFT) [21]. Note that our aim is not to compare segmentation methods, but to show that several of them can benefit from a disciplined, systematic, objective, and effective procedure for arc-weight assignment, followed by some proper approach-specific adaptive procedure, such as the complement of the weights, owing to the nature of the meaning of weights in some methods; or by some tuning procedure (e.g., non-maximal suppression [39], increasing transformations [39], gradient orientation (Section 4.7)). The visualization of the arc weights also allows the user to choose the most appropriate method for a given image. For example, it is desirable in live wire that the arc weights be lower along the object’s boundary than in the neighborhood around it [16], [22]; the local affinities in relative-fuzzy connectedness [47] be higher inside and outside the object than on its boundary; the gradient values in watershed transforms be higher for pixels on the object’s boundary than in its interior and exterior [32], [4], [15]; the gradient values in tree pruning be higher on the object’s boundary than in its interior, and, at least, in a neighborhood in its exterior [3]; and the arc weights in graph-cut segmentation be lower across the object’s boundary than in its interior and exterior [7], [51], [56]. Additionally, energy minimization in [7] using the min-cut/max-flow algorithm from source to sink nodes also requires higher arc weights between source and object pixels, lower arc weights between source and background pixels, lower arc weights between sink and object pixels, and higher arc weights between sink and background pixels. Clearly, the effectiveness of these approaches suffers when the above desirable conditions are not satisfied, and this explains why the visual feedback helps the user to choose the most appropriate method for a given segmentation task.
To outline this paper: Section 2 presents the basic concepts on image graphs and the terminology adopted in this paper. Arc-weight estimation is presented in Section 3 by showing how to exploit image attributes and object information provided by user-selected markers. Section 4 describes several interactive segmentation methods based on the arc-weight assignment of Section 3, and the main advantages of the synergistic approach are demonstrated in Section 5, including evaluation experiments with medical data from two imaging modalities (CT and MRI). Our conclusions are stated in Section 6.
Section snippets
Basic concepts on image graphs
An image is a pair where is the image domain and assigns a set of m scalars , to each pixel . This definition applies to multi-dimensional and multi-parametric images. For example, may be the red, green and blue values of s in a color image . The subindex i is dropped for gray images since it becomes awkward when .
An irreflexive adjacency relation is a binary relation between distinct pixels. We use or to
Synergistic arc-weight estimation
Arc-weight estimation takes into account image attributes and object information in order to enhance the discontinuities between object and background. Let v be an algorithm which extracts attributes (color, gradient, texture) from any pixel and returns a vector . In the simplest case, we may take . However, the best set of attributes depends on each given application.
Interactive segmentation methods
A segmentation result is represented by a label image , in which each label assigns a pixel to one object out of c objects, including background. For the sake of simplicity, we have considered the case of in all examples of this paper. All methods presented in this section have been well published, so we will present only a short description with their graph parameters customized as a function of and , although for other methods, different adaptive
Experiments and results
The examples in the previous sections have shown that the proposed process for arc-weight assignment is useful in several image segmentation methods. The synergism between the user and the computer offers some important advantages as well. Object information is incorporated into arc-weight estimation under user supervision and control. In traditional segmentation methods [28], [4], [51], [56], arc-weight estimation is usually treated as a simple embedded process, disregarding, in many cases,
Conclusion
We have presented an interactive method for arc-weight estimation, which can be employed effectively by several graph-based segmentation approaches as we demonstrated. Our method exploits in a synergistic way the human abilities for recognition and the computer abilities for delineation. While the user draws markers inside each object (including background), arc weights are estimated from image attributes and object information (pixels under the markers), and a visual feedback guides the user’s
Acknowledgments
The authors thank FAPESP and CNPq for the financial support. The images of Fig. 3, Fig. 4, Fig. 6, Fig. 10, Fig. 14 have been obtained from http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping.
References (59)
- et al.
Three-dimensional segmentation of anatomical structures in MR images on large data bases
Magnetic Resonance Imaging
(2001) - et al.
Iterative relative fuzzy connectedness for multiple objects with multiple seeds
Computer Vision and Image Understanding
(2007) - et al.
User-steered image segmentation paradigms: live-wire and live-lane
Graphical Models and Image Processing
(1998) - et al.
Multiscale skeletons by image foresting transform and its applications to neuromorphometry
Pattern Recognition
(2002) - et al.
Live-wire-based segmentation using similarities between corresponding image structures
Computerized Medical Imaging and Graphics
(2007) G-wire: a livewire segmentation algorithm based on a generalized graph formulation
Pattern Recognition Letters
(2005)- et al.
Enhanced lane: interactive image segmentation by incremental path map construction
Graphical Models
(2002) - et al.
Automatic segmentation of left and right cerebral hemispheres from MRI brain volumes using the graph cuts algorithm
NeuroImage
(2007) - et al.
Interaction in the segmentation of medical images: a survey
Medical Image Analysis
(2001) - et al.
Relative fuzzy connectedness among multiple objects: theory, algorithms, and applications in image segmentation
Computer Vision and Image Understanding
(2001)
Contour salience descriptors for effective image retrieval and analysis
Image and Vision Computing
A framework for evaluating image segmentation algorithms
Computerized Medical Imaging and Graphics
Automatic image segmentation by tree pruning
Journal of Mathematical Imaging and Vision
The morphological approach to segmentation: the watershed transformation
An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision
IEEE Transactions on Pattern Analysis and Machine Intelligence
Fast approximate energy minimization via graph cuts
IEEE Transactions on Pattern Analysis and Machine Intelligence
Design and construction of a realistic digital brain phantom
IEEE Transactions on Medical Imaging
Introduction to Algorithms
Measures of the amount of ecologic association between species
Ecology
Pattern Classification
Interactive volume segmentation with differential image foresting transforms
IEEE Transactions on Medical Imaging
The image foresting transform: theory, algorithms, and applications
IEEE Transactions on Pattern Analysis and Machine Intelligence
Cited by (54)
Grabber: A tool to improve convergence in interactive image segmentation
2020, Pattern Recognition LettersCitation Excerpt :Initially, the methods did not learn the parameters of the segmentation model [2,3]. Many methods then started learning a model but fixing it during segmentation [1,4–7]. Other approaches can learn and update a model from user inputs during segmentation [8–12] and several recent ones start from a pre-trained model that is adapted from user inputs during segmentation [13–15].
Gradient-based generation of intermediate images for heterogeneous tumor segmentation within hybrid PET/MRI scans
2020, Computers in Biology and MedicineHybrid PET/MRI co-segmentation based on joint fuzzy connectedness and graph cut
2017, Computer Methods and Programs in BiomedicineUnsupervised hierarchical image segmentation through fuzzy entropy maximization
2017, Pattern RecognitionCitation Excerpt :The third group is to combine threshold-based segmentation results with graph cuts (GC). By setting additional smoothness term, the spatial coherence can be enforced [29,39,40]. It is difficult to apply fuzzy c-partition entropy to color image segmentation directly.
Multiscale 2D medial axes and 3D surface skeletons by the image foresting transform
2017, Skeletonization: Theory, Methods and ApplicationsCooperative pose estimation of a fleet of robots based on interactive points alignment
2016, Expert Systems with Applications