Model-based graph-cut method for automatic flower segmentation with spatial constraints

Highlights

• An improvement of the classical energy function (graph-cut) is proposed.

• Integration of spatial priori and gradient information improves segmentation result.

• A new “coarse-to-fine” flower segmentation method is presented and implemented.

Abstract

In this paper, we present an accelerated system for segmenting flower images based on graph-cut technique which formulates the segmentation problem as an energy function minimization. The contribution of this paper consists to propose an improvement of the classical used energy function, which is composed of a data-consistent term and a boundary term. For this, we integrate an additional data-consistent term based on the spatial prior and we add gradient information in the boundary term. Then, we propose an automated coarse-to-fine segmentation method composed mainly of two levels: coarse segmentation and fine segmentation. First, the coarse segmentation level is based on minimizing the proposed energy function. Then, the fine segmentation is done by optimizing the energy function through the standard graph-cut technique. Experiments were performed on a subset of Oxford flower database and the obtained results are compared to the reimplemented method of Nilsback et al. [1]. The evaluation shows that our method consumes less CPU time and it has a satisfactory accuracy compared with the mentioned method above [1].

Introduction

Automatic flower classification systems are important for a wide range of application including pharmacy research, environment protection and perfume production. Thanks to computer vision, image processing and pattern recognition techniques, automatic recognition systems make the identification of the flower category easier by analyzing color images. Image segmentation is generally considered an important component of the recognition or classification processes, and affects the quality of the image analysis. Automatic flower segmentation allows the extraction of the object of interest (foreground) from the rest of the image (background) without any manual interaction.

The majority of flower images present natural scenes with complex background. The areas surrounding the flowers have generally large variety of colors and textures. It can contain several entities distributed separately or together such as stones, leaves, turf grass, green foliage and soil. Fig. 1 illustrates different types of elements that can be contained in the area surrounding the flower. As the flowers from different species may look very similar both in shape and color, the use of the background information to generate the image features can increase this similarity and consequently reduce the classification accuracy.

Therefore, we believe that the extraction of features only from the object of interest provides more meaningful and accurate information than the one obtained from the whole image. Although many flower image segmentation methods have been proposed in the literature, it remains difficult to find a general solution that is applicable to all types of flowers and gives accurate results. In the next paragraph, we present the state of the art on flower image segmentation.

Das et al. [2] proposed an iterative segmentation algorithm using color and spatial domain knowledge-driven feedback. Their method mapped the RGB color space to commonly used color names in order to delete pixels which belong to background color classes like black, brown, green or gray. The foreground region represented by the remaining colors is accepted if it is included in the flower area. In order to define the flower region, some hypotheses were made such as the flower centroid should fall within the “central region” of the image. Saitoh et al. [3] presented the Normalized Cost (NC) method to extract flower regions. It is based on a Dynamic Programming method known as the intelligent scissors [4] for extracting the boundary of the object of interest. The image is represented as a directed weighted graph where nodes are pixels and arcs between neighboring pixels represent the 8-connectivity information. This method consists in computing the local minimum cost given by a path between two seeds. The obtained cost is normalized by the length of this path. The shortest path in the graph gives the object edges. In this work, the authors assume that the flower is at the center of the image and the background occupies the peripheral area. Based on this hypothesis, the authors determine some local minimum points of each local cost profile along the straight line from the starting point to all the middle points of four sides. Then, they extract the boundary for each local minimum point based on the NC and they select the one that has the smallest normalized cost and contains the center point. Another interesting automatic algorithm can be found in [1]. The first step of this algorithm aims to obtain an initial flower segmentation using the contrast dependent prior MRF (Markov Random Field) cost function, which is optimized by using graph-cut [5] based on general foreground and background color distributions. These distributions are learned by labeling pixels in few training images for each flower class as foreground or background. Then, these distributions are averaged over all classes. A generic flower-shaped model is then fitted to this initial segmentation in order to detect petals which have a loose geometric consistency using an affine invariant Hough-like procedure. The image region corresponding to the petals is used to obtain image-specific foreground and background color distributions which will be employed to obtain new color distributions by blending them with general ones. A new segmentation using MRF-based graph-cut is repeated using these new color distributions. The shape-model fitting and the segmentation will be iterated until convergence. In [6], Aydin and Ugur proposed a flower image segmentation algorithm based on ant colony optimization (ACO) [7]. First, RGB color space is converted to CIE-LAB color space in order to provide accuracy and perceptual approach in color difference determination. Second, the histogram of each color component is built (n bins for each component) to determine the center points of each bin and three “bin-centers” are obtained to generate all possible combinations of center points (n³) which constitute the potential candidate cluster's center points. Then, ant colony optimization algorithm is applied to select optimum cluster center points. Finally, each pixel is classified in the nearest cluster. Recently, Fukuda et al. [8] proposed an automatic flower segmentation model based on graph-cut. They set the region penalty in graph-cut based on posterior probability, and not on likelihood as in the classical graph-cut. The posterior probabilities are determined by using AdaBoost [9] and saliency map [10]. In fact, the AdaBoost classifier provides a rectangular window representing the approximate flower location used to compute foreground and background color distributions (likelihoods). Then, the saliency map is used to provide prior probabilities. Using the Bayes' theorem, posterior probabilities are computed and integrated in graph-cut. More recently, Chai et al. in [11] addressed the co-segmentation of flowers which consists of segmenting a similar object (flower) from a pair of images. Their algorithm iterates at two levels: segmenting each image independently with GrabCut [12] at the pixel level and classifying the obtained superpixels into foreground and background using SVM learning algorithm.

To conclude, all of these approaches aim the automatic segmentation of flower images but there are some differences between them in terms of quality and time-consuming. In fact, the approach proposed by [2] doesn't extract all flower regions in the image, but isolates a region as the best description of the flower color. Besides, the fact of eliminating pixels belonging to non-flower color classes will fail with the images containing brown or gray flowers which will be considered in the evaluation of our algorithm. All these factors can reduce the segmentation performance of this method [2]. As for the method proposed by [3], the main disadvantage is the high computational time due to the research of the local cost paths. As demonstrated in [1], the method of Nilsback et al. achieves good performances (93% in quality measure [13]) but it is very slow to be run in real-time applications. In this method [1], fitting a generic shape model needs the determination of the corners, the petals and the center of the flowers which requires high computational time. The method proposed by [6] yields accurate segmentation (87% in quality measure) but we have no information about the processing time of the algorithm. The method reviewed by [11] achieves about 94% segmentation accuracy not on the original data split of the Oxford-17 flower dataset. As cited in [14], the algorithm proposed by iteChai11 needs more than 30s as run-time for segmentation. Finally, in [8], it is demonstrated that the proposed method can reduce the error rate in segmenting flower images with no mention of the consuming time. In terms of evaluation settings, all aforementioned flower segmentation methods have been performed on different datasets and evaluated with different accuracy measure as shown in Table 1.

After this study, one can say that the model proposed by Nilsback et al. [1] yields the best performance of automatic flower image segmentation even if it is very time-consuming. That is why we take it as a state-of-the-art in order to improve it or to propose a new fast algorithm offering similar accuracy.

Recently, it has been an important interest in image segmentation approaches based on graph-cut [15], [16], [17], [18]. Many works use graph-cut technique since it gives positive results on both medical and natural images [19], [12]. Unfortunately, the standard graph-cut algorithm suffers from some limits: (i) it fails to give a smooth segmentation result by labeling some object pixels as background, or vice versa. (ii) It fails to define the desired boundary of the object. In fact, if object pixels have color distribution similar to the image background then these pixels will be labeled as background, and they won't be considered as part of the object. This failure can be explained by the fact that the traditional energy function to be minimized uses only the color information. In order to alleviate these problems, some solutions have been proposed to integrate a priori information other than color information. It has been demonstrated that the segmentation results of graph-cut methods can be improved by introducing novel constraints in the segmentation process such as shape constraints [20], [21] and spatial constraints [22]. For example, authors in [21] proposed to incorporate a generic star-shaped prior in the energy function minimized by graph-cut. The shape prior is represented as an object mask form using the distance transform of the star shape. Although this method [21] gives encouraging results with the assumption that the center of the shape is given by the user, the extracted object shape tends to be star-aligned. While shape-based graph-cut works can be robust, they have a shortcoming to segment object with high shape variability [23].

In this paper, our aim is to modify the standard graph energy function to enhance the segmentation result. Therefore, we introduce spatial constraints through an additional term in the formulation of the graph-cut energy function. Thus, we obtain a better classification of pixels as foreground despite their high probabilities of belonging to the background if we consider only their colors and vice versa. We integrate also gradient information in the traditional graph energy function to better express boundary constraints. Consequently, true boundaries can be determined based on both gradient and color information. The rest of the paper is organized as follows: In Section 2, first, we present the mathematical theory of graph-cut technique and its use for energy minimization. Then, we introduce our proposed new energy function to be minimized by graph-cut. Finally, our automatic flower segmentation algorithm is described. In Section 3, we present the experimental results and the evaluation of the proposed segmentation method.