Semiautomatic segmentation with compact shape prior☆
Introduction
Segmentation is generally defined as the problem of partitioning an image into two or more constituent components, where each component has a short summary representation. This definition is rather vague, because general purpose segmentation is not well defined. Segmentation becomes a much better defined problem when it is developed for a particular application, since then one frequently has a clearer idea of the properties a segmentation should have.
There are mainly three approaches to segmentation: automatic, manual and interactive. Manual segmentation is labor extensive and extremely time consuming. Purely automatic segmentation is very challenging, due to ambiguities in the presence of multiple objects, image noise, weak edges, etc. Ambiguity problems can be eased with user guidance, which is the idea of interactive segmentation methods. Hence, their popularity is increasing in applications in different domains [18], [24], [5], [3], [23], [1], [4].
The motivation behind our work is to reduce interaction to the minimum, asking the user to just choose the object of interest by clicking inside it. We call our approach semiautomatic segmentation, to distinguish it from general interactive segmentation, where the user is allowed to provide a potentially unlimited amount of guidance. The name semiautomatic is used to emphasize that our algorithm is only a step away from the automatic segmentation, since only one seed point is required from the user. General interactive segmentation can be quite far from automatic segmentation if lots of input is required from the user in order to achieve satisfactory results.
To produce an accurate and robust segmentation, we have to develop our algorithm with some application in mind, since, as we have already mentioned, general purpose segmentation is an ill-defined problem. We chose to design our algorithm in the context of an interesting industrial application, which requires transistor gates to be segmented from the images of integrated chips.
Over the years, researchers have developed different techniques for segmentation. Some of the primitive methods that have been popular because of their simplicity are region growing, split-and-merge, edge detection and thresholding, see, for example, Gonzalez and Woods [15]. Although these methods and their variants are still widely used, they are not robust as they are based on local decisions. For example, the major problem with region growing is the “leaking” through weak points in the boundary, which is inevitable in most images. Likewise, thresholding fails when the object of interest is not homogeneous. In particular, objects with smoothly varying intensities are split into several segments.
To overcome problems due to local decision strategies, global properties have to be included in the segmentation. Graph theoretic approach to segmentation allows us to do so. Various graph based algorithms have been proposed over the years [33], [27], [30], [5], [17], [12], [32], [4], [16]. They differ in the way the segmentation is interpreted and in the techniques employed to solve the problem. However, all these methods typically involve two main steps – formulating an objective function and optimizing it.
In some approaches, such as live wire [11], [24], a global objective function is implicit. Live wire is a paradigm for segmentation that requires the user to mark a seed on the object boundary. As the user moves the cursor (the free point) close to the object boundary, a curve (livewire) clings to the object boundary and segments the object. The curve position is optimized by finding the shortest path on a certain graph. In this approach considerable amount of interaction may be required in order to find the appropriate segmentation.
Level sets sets [25], normalized cut [27], active contour (snake) evolution [18], [7], [2], and graph cut [5] formulate the energy function explicitly based on various global properties that the segmentation is expected to have. Unfortunately, for many energy functions that one may wish to formulate, finding their global minimum is computationally prohibitive. Normalized cut computes only an approximation to the global minimum, and in most cases, active contours and level sets compute only a local minimum (a few notable special case exceptions are Cohen and Kimmel [8], [21]).
The advantage of the graph cut compared to the above listed methods is that it guarantees a globally optimal solution for a family of energy functions. An additional benefit is that one can easily incorporate both regional and boundary properties of segmentation. Also, unlike most active contour/level set methods, graph cut is not sensitive to the initialization [4]. Furthermore, level sets/snakes would be unsuitable for our semiautomatic approach since they require the user to initialize a contour, not just one point. These advantages make the graph cut method much more attractive than others in achieving our goal.
As segmentation is a subjective problem, we start with the already mentioned application of transistor gate segmentation in the images of integrated chips. We make several assumptions based on the prior knowledge of our data and fit them into the framework of the algorithm in Boykov and Jolly [5]. The most important assumption that we make is that an object to be segmented is compact1 in shape. While this assumption allows us to produce very robust segmentations, it is also our most restrictive assumption, making our algorithm not suitable for segmentation of objects of general shapes. However, apart from the transistor gates there are important applications (industrial and medical) where the objects of interest are approximately compact. Furthermore, we can also handle objects with somewhat more general shapes, specifically the objects that can be divided either vertically or horizontally into several approximately collinear pieces, where each piece is compact in shape.
There are several related methods that incorporate shape priors into graph cut segmentation. In Slabaugh and Unal [28] the authors incorporate an elliptical prior in an iterative refinement process. The disadvantages of this approach is that it is iterative and the elliptic shape assumption is overly restrictive for many applications. In Freedman and Zhang [14], the shape prior can be arbitrary, but their method requires a very accurate registration of the assumed shape with the actual location of the object of interest in the image, which is a difficult task in itself. In Kumar et al. [20], they also require fitting of a model of a certain shape to an image, and their method, which uses sampling for estimation of model’s parameters, is very computationally intensive.
The use of shape priors for segmentation has been investigated before. Recently there has been a lot of work on using shape priors in level set segmentation, some examples are Leventon et al. [22], Tsai et al. [29], Rousson and Paragios [26], Cremers et al. [10], Cremers et al. [9]. However, level set segmentation is not numerically stable and the solution is prone to getting stuck in a local minimum.
Another issue that we address is the parameter selection. In the framework of Boykov and Jolly [5], the values of parameters have a direct impact on the result produced by the algorithm. Unfortunate choice of parameters can produce unacceptable segmentation results that have to be detected by the user and corrected by possibly a considerable amount of interaction. This is not acceptable for our semiautomatic approach, since our goal is to reduce user interaction to a single click. If the segmentation algorithm is used for a collection of images that do not exhibit large variability, then it is possible to select the parameters that work well for that type of images beforehand. However, we found that for our application, the images do exhibit considerable variability and selecting fixed parameters that work well for most instances is not possible. For each image, there is an optimal setting of parameters that works well, but estimating that range is difficult. Our solution is to run the segmentation algorithm for a range of parameters and choose the highest quality segmentation. This, of course, requires some way of judging the quality of segmentation. We devise a simple but intuitive test to check the quality of the segment automatically. This “quality check” is application dependent. If the current segment does not pass the quality check, the parameters are readjusted and the graph cut step is redone with the new parameters. We iterate this process using a search over parameter space until the resulting segment passes the quality check. Thus in our work, we estimate all the important parameters of the algorithm automatically.
If we could directly incorporate our”quality check” into the energy function, then we would not have to search over a range of parameters but could compute the best quality segment in one step. Unfortunately we cannot incorporate our quality check into the energy function in such a way that it still can be minimized with a graph cut.
When the user provides many seed points, or when an accurate color model of the object of interest is known, the regional properties of the object can be relied on, and are included in the graph cut segmentation with a large weight. Our goal is to have a very low input from the user, who just marks one object seed point. Thus we do not have enough samples from the user to construct a reliable model for the color distribution of the object. In this case we have to allow the object to deviate from the unreliable color model, and therefore the regional terms are given a smaller weight (the smaller the weight of the regional terms, the more is the object allowed to deviate from the color model). When regional terms have smaller weight, boundary terms become relatively more important. It makes sense intuitively, since if there is no reliable color model, we must rely more on the fact that we expect the object boundary to aligns with intensity edges in the image. A serious difficulty in graph cut segmentation in the case when regional terms have a small weight is that there is a bias towards producing segments with shorter boundaries. In our framework, we can easily counteract this bias. It turns out that due to incorporating compact shape prior in the graph cut framework, we can introduce a new parameter bias, which biases the algorithm towards a larger object segment.2 The bias is exactly the parameter for which we search over a range of values to find the segmentation that passes the quality check mentioned above.
Thus our main contributions to the graph cut segmentation framework of Boykov and Jolly [5] are as follows. We introduce the idea of an application dependent “quality check” which can be effectively used for automatic parameter selection. We introduce the compact shape prior, which lets us deal with the objects of compact shape very robustly. Lastly, due to the shape prior, we are able to introduce a bias parameter which allows us to counteract the shrinking bias of the graph cut segmentation.
We evaluate our approach on a transistor segmentation application for Semiconductor Insights, which is an engineering consultancy company specializing in intellectual property protection and competitive intelligence in the integrated circuit domain. Our segmentation algorithm produces highly accurate results in real-time,3 and was used to upgrade their manual system to a semiautomatic one.
This paper is organized as follows. In Section 2, we review the graph cut segmentation framework of Boykov and Jolly [5], in Section 3 we describe our work, in Section 4, we present our experimental results and we finally conclude with a discussion in Section 5.
Section snippets
Graph cut segmentation
In this section we briefly review the graph cut segmentation algorithm in Boykov and Jolly [5].
Our work
The goal of our semiautomatic segmentation is accurate and robust segmentation with user interaction restricted to a single click inside the object of interest. The graph cut algorithm [5] has several issues which make its direct use unsuitable for semiautomatic segmentation. We address these issues in our work.
In Boykov and Jolly [5], the user has to initially select a few object and background seeds. After running the algorithm the user has to inspect the quality of the segmentation. If
Results
We explored the challenging industrial problem of transistor gate segmentation in the images of integrated chips. It is an important preliminary step for performing intellectual property protection and competitive intelligence analysis in integrated circuitry domain. To obtain the images, the integrated circuit is de-layered and SEM micro-photographed. The images of the upper layers of the chip, that contain the metal wiring, are typically of high quality and can be segmented by automated
Discussion
In this paper, we presented a semiautomatic segmentation algorithm developed by modifying the basic graph cut segmentation algorithm of Boykov and Jolly [5]. We showed how problem specific assumptions and constraints can be well utilized to reduce the user interaction and also the complexity of the problem. The main contribution of our work is the introduction of the compact shape prior into the graph cut segmentation, which adds robustness to the algorithm. An additional benefit of using the
Acknowledgements
We thank Stephen Begg and Dale Carlson of Semiconductor Insights for developing interactive application incorporating the method.
References (33)
On active contour models and ballons
Computer Vision, Graphics and Image Processing
(1991)- et al.
Fronts propagating with curvature dependent speed: Algorithm based on hamilton jacobi formulations
Journal of Computational Physics
(1988) - A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D. Salesin, M. Cohen, Interactive digital...
- et al.
Using dynamic programming for solving variational problems in vision
(1990) - A. Blake, C. Rother, M. Brown, P. Perez, P. Torr, Interactive image segmentation using an adaptive gmmrf model. in:...
- et al.
Graph cuts and efficient n-d image segmentation
International Journal of Computer Vision
(2006) - Y. Boykov, M.P. Jolly, Interactive graph cuts for optimal boundary and region segmentation, in: International...
- et al.
An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision
IEEE Transaction on PAMI
(2004) - L.D. Cohen, R. Kimmel, Global minimum for active contour models: a minimal path approach, in: IEEE Conference on...
- D. Cremers, Nonlinear dynamical shape priors for level set segmentation, in: IEEE Conference on Computer Vision and...
Kernel density estimation and intrinsic alignment for shape priors in level set segmentation
International Journal of Computer Vision
Efficient graph-based image segmentation
International Journal of Computer Vision
Flows in Networks
Digital Image Processing
Cited by (64)
Automatic MPST-cut for segmentation of carpal bones from MR volumes
2017, Computers in Biology and MedicineCitation Excerpt :The only interaction required is the selection of a source point inside the object of interest. In contrast, conventional methods require several user interactions, e.g., the setting of several parameters and thresholds, which can result in a lack of robustness, poor reproducibility, and suboptimal results [8]. In addition, unlike other published approaches that employ a priori information, such as atlas or shape [9], the proposed technique does not require a priori model or knowledge and is adaptive to the data.
Joint optimization of segmentation and shape prior from level-set-based statistical shape model, and its application to the automated segmentation of abdominal organs
2016, Medical Image AnalysisCitation Excerpt :Most conventional studies based on graph cuts have employed the first strategy, and a variety of shape priors have been proposed. General shape constraints, such as ellipse (Slabaugh and Unal, 2005), blob-like (Funka-Lea et al., 2006), or compact (Das et al., 2009) priors, have been successfully applied to the segmentation of a wide range of objects. While these shape priors are useful, they may be too simple for segmenting objects with more complex shapes.
Semi-automatic Extraction and Regularization of Buildings of Different Shapes from High-Resolution Remote Sensing Images
2022, Yingyong Kexue Xuebao/Journal of Applied SciencesSSP-REGULARIZER: A STAR SHAPE PRIOR BASED REGULARIZER FOR VESSEL LUMEN SEGMENTATION IN OCT IMAGES
2022, Proceedings - International Conference on Image Processing, ICIPREFICS: A Step Towards Linking Vision with Hardware Assurance
2022, Proceedings - 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022Hardware Trust and Assurance through Reverse Engineering: A Tutorial and Outlook from Image Analysis and Machine Learning Perspectives
2021, ACM Journal on Emerging Technologies in Computing Systems
- ☆
This research was partially supported by NSERC and Semiconductor Insights.