1 Introduction

Superpixel computation is the process of grouping similar pixels trying to preserve image boundaries. Some of the methods to compute superpixels can be seen as image segmentation methods and can be done by hierarchical strategies [11]. A hierarchical image segmentation is a set of image segmentations at different detail levels that preserves spatial and neighboring information among the segmented regions. Hierarchies of partitions have been used in many approaches in the context of image and video segmentation [3, 10, 12, 20,21,22,23, 25]. Following the idea of [27], we propose a supervised image segmentation based on the label propagation guided by hierarchies of partitions taking into account information about the values extracted from the hierarchy as area and level value. The seminal idea for label propagation on a hierarchy of segmentations was proposed in [27], and later used for hierarchical segmentation assessment in [21, 22]: in this approach, the labels from the background and from the foreground compete with each other in order to label the undefined regions. Differently of [21, 22, 27], in this work, we propose a method for label propagation in which different criteria are used to decide which label must be propagated. The general strategy can be divided into three main steps: (i) automatically generate seeds (or markers); (ii) compute a hierarchy of partitions taking into account a specific attribute, such as the area or the volume, in which the pixels are represented on the leaves; and (iii) from leaves to root on the hierarchy, the labels are propagated considering a competition strategy. To decide which of the competing labels must be propagated into a neighbor region in the case of conflict, we have devised two different strategies: (i) the label that is related to a region with the highest attribute value is propagated; or (ii) the label that is related to the region with the smallest attribute value is propagated. It is worth to mention that some attribute values may directly be extracted from the hierarchy such as the hierarchy level, but others must be dynamically computed, i.e. the area of the regions.

This work is organized as follows. In Sect. 2, we present some important concepts related to superpixels computation and hierarchies. The proposed label propagation guided by hierarchy of partitions is described in Sect. 3. Some experimental results obtained for tests performed on four databases are detailed in Sect. 4. Finally, in Sect. 5, some conclusions are drawn and further works are discussed.

2 Fundamental Concepts

2.1 Graphs

A graph is as a pair (V, E) such that V is a finite set and E is composed of unordered pairs of distinct elements in V, i.e., E is a subset of \( \{\{x,y\} \subseteq V \mid x \not = y\}.\) In the following, the pair \(G=(V,E)\) denotes a connected graph. Each element of V is called a vertex or a pixel, and each element of E is called an edge. The graph G is used to model the image domain, e.g., V will represent a regular 2D grid of pixels, and E encoded the 4 adjacency for every pixel. We denote by W a function from E to \(\mathbb {R}\) that weights the edges of G. Therefore, the pair (G, W) is an edge-weighted graph, and, for any \(u \in E\), the value W(u) represents the weight of u.

2.2 Hierarchies

A partition, also called a segmentation, \(\mathbf P \) of V is a family of subsets of V such that: (i) the intersection of any two distinct elements of \(\mathbf P \) is empty; and (ii) the union of the elements of \(\mathbf P \) is equal to V. Each element of a partition \(\mathbf P \) is called a region of the partition \(\mathbf P \). Given two partitions \(\mathbf P _{1}\) and \(\mathbf P _{2}\), we say that \(\mathbf P _{2}\) is a refinement of \(\mathbf P _{1}\) if every region of \(\mathbf P _{2}\) is included in a region of \(\mathbf P _{1}\).

A hierarchy (of partitions) \(H = (\mathbf P _{0},...,\mathbf P _{n})\) is a sequence of partitions of V such that \(\mathbf P _{0}\) is the single region partition \(\mathbf P _{0} = \{V\}\), the partition \(\mathbf P _{n}\) contains every singletons of V, i.e., \(\mathbf P _{n} = \{\{x\} \mid x \in V\}\), and \(\mathbf P _{i}\) is a refinement of \(\mathbf P _{i-1}\) for all i in {1, ..., n}. It is usually represented as a tree or a dendrogram and can be visualized as a saliency map, which is a contour map in which the grey level represents the strength of the contour: i.e., its level of disappearance in the hierarchy. The reader may refer to [7] for more details about the hierarchy of partitions.

The Quasi-Flat Zones (QFZ) hierarchy is a classical structure that is constructed by considering the connected components of the level sets of the dissimilarity function [7, 16, 17]. More precisely, we say that two pixels are \(\lambda \)-connected if they can be joined by a path such that the dissimilarity between any two successive pixels of this path is lower than \(\lambda \). For a given \(\lambda \) in \(\mathbb {R}\), the equivalence classes of the relation “is \(\lambda \)-connected” form the \(\lambda \)-partition of the image into its \(\lambda \)-connected components also called \(\lambda \)-flat zones. The set of all \(\lambda \)-partitions for every \(\lambda \) in \(\mathbb {R}\) forms the QFZ hierarchy.

Watershed (WS) hierarchies are constructed by considering the watershed segmentation of an image that is iteratively flooded under the control of an attribute [4, 6, 8, 15, 18, 19, 21]. For example, the watershed segmentations of the area closings of size k of an image for every positive integer k form the WS hierarchy by area of the image. In this article, we consider 2 possible attributes: (i) area (WSArea); and (ii) volume (WSVol).

2.3 Seed Location and Superpixel Computation

According to [28], in order to really develop a superpixel approach one should address some requirements such as being a partition of the image, representing connected sets of pixels, preserving image boundaries, being generated efficiently and with a controllable number of generated superpixels. Usually, the methods for superpixel computation are classified in six different types: (i) Watershed-based [14]; (ii) Density-based [1]; (iii) Graph-based [24]; (iv) Contour evolution [5]; (v) Clustering-based [1]; and (vi) Boundary-aware.

In the context of superpixels, computer vision applications have come to rely increasingly attention on superpixels computation mainly due to the SLIC method [1]. The SLIC algorithm groups pixels into regions which can be used to replace the rigid structure of the pixel grid. Firstly, seeds are distributed throughout the image taking into account uniform regions, each one containing a centroid seed. The approximate size of each superpixel is N / K pixels, where N is the size of the image and K is the number of regions (or superpixels). Thus, seeds are equally spaced in the range of \(S = \sqrt{N/K}\). During the execution of the SLIC algorithm, the centers of the superpixels are moved to the lowest gradient position in a \(3\times ~3\) neighborhood to avoid starting at a very noisy pixel. Next, in the assignment step, each pixel is associated to the nearest cluster center according to a distance measure D. Thereafter, an update step, based on the K-means clustering algorithm, to compute the centers of the superpixels is done. The assignment and update steps are then repeated until stability or up to the maximum number of iterations. The strategy of seed location based on the SLIC method is so-called grid sampling. Despite the good results obtained by this sampling strategy, we can use a random sampling in order to randomly put seed on the image.

3 Label Propagation Guided by a Hiearchy of Partitions

The seminal idea for label propagation on a hierarchical watershed, was proposed in [27] in which labels from background and foreground compete with each other in order to be assigned to undefined regions. Differently of [27], in this work, we propose a method for label propagation in which distinct criteria may be used to decide which label must be propagated. The proposed strategy may be divided into three main steps: (i) seed generation; (ii) computation of the hierarchy of partitions; and (iii) label propagation.

In the first step, seed generation, seeds (or markers) are automatically distributed over the pixels. For computing the hierarchy of partition, we follow [8, 19] in order to compute a QFZ hierarchy and watershed hierarchies based on some attributes: (i) area; and (ii) and volume. The underlying graph corresponds to a 4-connected adjacency graph in which each edge is weighted by the dissimilarity measure between two pixels.

figure a

Finally, a label propagation is done following the procedure described in the Algorithm 1. The propagation is divided into two main steps: (i) from leaves to root (lines 3–9) to assign labels to the regions; and (ii) from root to leaves (lines 10–14) to assign labels to the pixels. In the first part, from leaves to root on the hierarchy, labels are propagated considering a competition strategy. In order to decide about which label must be propagated to a neighbor region in the case of conflict (line 4), we have used two different strategies (\(\mathbf {selectBestLabel}(S,\mathcal {A},\mathcal {C})\)): (i) the label that is related to a region with highest attribute value is propagated; or (ii) the label that is related to a region with smallest attribute value is propagated. In the second part, from root to leaves, the pixels of the regions receive the labels which were assigned to it during the bottow-up procedure.

Fig. 1.
figure 1

Toy example for illustrating the label propagation taking into account two different criteria for solving the seed competition, the higher value and higher area.

Full size image

It is worth to mention that some attributes may be directly extracted from the hierachy, that could be represented as a tree, such as the value of the hierarchy level, but other measures must dynamically be computed over the tree, i.e. area of the regions. In Fig. 1, we present a toy example for illustrating the label propagation procedure. In this example, we have used a QFZ hierarchy and we put three different seeds on the leaves Fig. 1(a). First, a bottom-up approach is made in which the labels are propagated from the leaves to the root. If there are label conflicts like in Figs. 1(b-left) and (c-left), the attributes computed from the hierarchy are taken into consideration for deciding which label must be propagated. For instance, in Figs. 1(b), the propagation is based on the smallest value of the hierarchy level, while in Figs. 1(b) it is based on the largest region area. As illustrated in Fig. 1(b-right) and (c-right), the obtained results are quite different. As can be observed, the leftmost part of the hierarchy are either assigned to yellow label or blue label. This occurs due to the criterion used to solve the conflict in the inner node of the tree.

4 Experimental Results

In this section, we present assessments in terms of undersegmentation error, boundary recall and GT Covering by using three hierarchies of watershed and two different gradients applied to three different image databases.

4.1 Experimental Setup and Datasets

In order to provide a comparative analysis between several strategies, we have used three different databases: (i) the Berkeley Segmentation Dataset [13], called BSDS500; (ii) the database proposed in [26], called GRABCUT; and (iii) the database proposed in [2] which is divided into two groups – single and two objects – called WI1OBJ and WI2OBJ, respectively. Here, we compare our method with several criteria to SLIC, which is the baseline of this work.

We considered a 4-adjacency relation with a Lab gradient (that is the Euclidean distance in the L*a*b* colour space) for the dissimilarity measure or max value of the structured edge gradient [9]. We have also studied two different strategies for seed generation, the grid sampling and the random sampling.

In this paper, we have used the following notation for simplification: \(\mathcal {A}^\downarrow _\mathcal {C}\) (or \(\mathcal {A}^\uparrow _\mathcal {C}\)) in which \(^\downarrow \) (\(^\uparrow \)) means the smallest (highest) value for the used criterion, \(\mathcal {A}\) can be the QFZ hierarchy (Q), watershed by area (A) and watershed by volume (V), and the \(\mathcal {C}\) can be the level value (v) and the area (a).

4.2 Quantitative and Qualitative Analysis

For a quantitative assessment, we illustrate in Figs. 2 and 3 several results computed over BSDS500, GRABCUT, WI1OBJ and WI2OBJ in terms of undersegmentation errors, boundary recall and GT covering. Due to lack of space, we present the results for structured edges as gradient since they outperform in all measures the Lab gradient.

Fig. 2.
figure 2

Evaluation of superpixel computation using grid sampling. For hierarchical watersheds, we have used SE and Euclidean Distance as gradient. From top to bottom: (i) BSDS500; (ii) Grabcut; (iii) Weizmann 1 object; and (iv) Weizmann 2 objects.

Full size image
Fig. 3.
figure 3

Evaluation of superpixel computation using random sampling. For hierarchical watersheds, we have used SE and Euclidean Distance as gradient. From top to bottom: (i) BSDS500; (ii) Grabcut; (iii) Weizmann 1 object; and (iv) Weizmann 2 objects.

Full size image

In a general way, the strategies with the highest value for solving the conflict have better GT Covering than the smallest value, but worse undersegmentation error and boundary recall. In our opinion, the label propagation guided by watershed by volume using the value of the level for solving the conflict outperforms other strategies in all databases. In particular, our strategies always outperform the SLIC for GT Covering. Regarding the seed sampling, there is no relevant difference in terms of the used measures between grid sampling and random sampling.

In Fig. 4, we illustrate some results obtained by our label propagation strategy. For computing the superpixels, we have used QFZ hierarchy Fig. 4(a) and Watershed by area Fig. 4(b). As we can see in these results, the superpixels computed from QFZ hierarchy are quite small. Considering that our strategy may produce disconnected regions, we can observe that despite that the number of seeds is defined, for example, in 8, the result presents more superpixels. Regarding the results for the watershed by area, we results are interesting since they better preserve the borders.

Fig. 4.
figure 4

Examples of label propagation. For computing the superpixels, we have used QFZ hierarchy (a) and Watershed by area (b). In order to devise about the conflict, we have used smallest level value of the hierarchy among the sons of a node (first row of each subfigure), and we have used smallest area related to the sons of a node (second row of each subfigure). Moreover, from left to right the used gradient and the number of superpixels are the following: L*a*b* with 8 seeds; SE with 8 seeds; L*a*b* with 128 seeds; and SE with 128 seeds.

Full size image

5 Conclusions

In this paper, we have proposed a label propagation method guided by hierarchy of partitions for superpixel computation. The proposed strategy may be divided into three main steps: (i) seed generation; (ii) computation of the hierarchy of partitions; and (iii) label propagation. The main novelty of this work is related to the use of different criteria to decide about which of competing label should be propagated to neighbor region. According to our experiments, the proposed method outperforms the SLIC method, which is the baseline. Moreover, the propagation based on watershed by volume using the smallest hierarchy level as criterion to solve the conflict outperform the other tested criteria and hierarchy. At the end of our procedure, the same label could be assigned to disconnected regions. We have solved this problem by creating labels to be assigned to each disconnected region. As future works, we will study how to produce connected regions at the end of our method. Furthermore, we intend to apply other criteria for solving the conflict and other hierarchies such as the watershed by number of nodes.