An evaluation of the compactness of superpixels
Introduction
Image segmentation is a fundamental task in computer vision and many applications rely on it as a preprocessing step. Superpixel segmentation belongs to the class of oversegmentation algorithms and the term superpixel was introduced by Ren and Malik (2003).
A superpixel is defined as a homogeneous image region that aligns well with object boundaries. This allows to represent an image with only a couple of hundred segments instead of tens of thousands of pixels. This reduction of input complexity makes superpixels particularly useful for a wide range of application domains, for example image segmentation (Achanta et al., 2010, Schick and Stiefelhagen, 2011, Veksler et al., 2010), object recognition (Achanta et al., 2010), object localization (Fulkerson et al., 2009), labeling tasks (Kohli et al., 2009), motion segmentation (Ayvaci and Soatto, 2009), foreground segmentation (Schick et al., 2012), tracking (Wang et al., 2011), and pose estimation (Mori, 2005, Mori et al., 2004), to name just a few.
A compact superpixel has a regular shape with smooth boundaries and many authors agree that compactness is desirable for superpixels (Achanta et al., 2010, Levinshtein et al., 2009, Liu et al., 2011, Moore et al., 2010, Veksler et al., 2010, Zeng et al., 2011, Zhang et al., 2011, Perbet and Maki, 2011). However, the compactness of superpixel segmentations has not yet been systematically measured and evaluated. We are the first to measure the compactness of superpixels and investigate its implications.
This work is an extended version of Schick et al. (2012). The main additional contribution is an extension of the superpixel segmentation in Schick et al. (2012) that guarantees that it conforms to a lattice. Further, we extended the experimental section with more discussions about compactness including a correlation, convergence, and lattice stability analysis.
The remainder of this paper is organized as follows. Related work is discussed in Section 2. The compactness metric is presented in Section 3. Section 4 introduces the segmentation algorithm followed by the proposed lattice constraints in Section 5. The evaluation is presented in Section 6 with results in Section 7. Section 8 demonstrates the benefits of compactness with an example application before the conclusion in Section 9.
Section snippets
Related work
Superpixels have received increasing attention in the last years and there is a wide range of superpixel segmentation algorithms. These algorithms differ in how they solve the segmentation task which results in different properties regarding runtime, segmentation quality, and superpixel shape. Graph-based segmentation algorithms were proposed by Shi and Malik (2000), a normalized cut approach by Malik et al. (2001), and a graph cut approach by Veksler et al. (2010). Zhang et al. (2011) proposed
Superpixel compactness
The larger the area of a shape for a given boundary length, the higher is its compactness. The same holds for superpixels and there seems to be an intuitive understanding that compactness is indeed a desirable property (Achanta et al., 2010, Levinshtein et al., 2009, Liu et al., 2011, Moore et al., 2010, Veksler et al., 2010, Zeng et al., 2011, Zhang et al., 2011, Perbet and Maki, 2011). We will now discuss why compactness is indeed advantageous before explaining the compactness metric.
Superpixel segmentation
In this section, we propose a modification of SLIC (Achanta et al., 2010) that computes more accurate superpixels with a transparent control of their compactness. (This algorithm was also presented in Schick et al. (2012).) SLIC is based on an iterative k-means clustering and computes superpixels utilizing both a distance in color space as well as Euclidean space. While the k-means algorithm leads to very accurate clusters, it does not guarantee that the clusters remain connected which is
Superpixel lattice
In this section, we present a modification of the algorithm proposed in Section 4 that additionally guarantees that the superpixels conform to a regular lattice. We first discuss the advantages of superpixel lattices before introducing the lattice constraints.
Evaluation
We compared the proposed algorithm to five different superpixels algorithms. The superpixels algorithms are the normalized cuts segmentation from Mori, 2005, Mori et al., 2004 (NC), SLIC (Achanta et al., 2010), TurboPixels (Levinshtein et al., 2009) (TP), entropy rate superpixels (Liu et al., 2011) (ERS), and Superpixel lattices (Moore et al., 2008) (LATTICE). SLIC allows to weight its Euclidean distance term which we included in the evaluation by testing the extreme values (SLIC_min, SLIC_max)
Results
This section presents the evaluation results. We first discuss quantitative results of all four metrics before presenting our compactness-specific findings. After discussing the impact of the lattice constraints on performance, we conclude with a set of images for a more qualitative impression.
Application
We showed in Section 1 that superpixels are applied to a wide range of different applications. Which superpixel properties are most important depends on the application in question. For some applications, the shape might not be important, e.g. when computing mean color values or histograms. For applications, however, that work directly with superpixel boundaries or that require a regular shape (Mori, 2005, Mori et al., 2004), the superpixel compactness might be very important. We will now
Conclusion
Compactness is an important and desirable property of superpixels (Achanta et al., 2010, Levinshtein et al., 2009, Liu et al., 2011, Moore et al., 2010, Veksler et al., 2010, Zeng et al., 2011, Zhang et al., 2011, Perbet and Maki, 2011). With this work, we proposed a metric to measure the compactness of superpixel segmentations. We further discussed implications and showed that there is a negative correlation between boundary recall and compactness. The advantages of controlling the compactness
Acknowledgments
This work was partially supported by the FhG Internal Programs under Grant No. 692026 and by OSEO, French State agency for innovation, as part of the Quaero Programme.
References (32)
- Achanta, R., Smith, K., Lucchi, A., Fua, P., Ssstrunk, S., 2010. SLIC Superpixels. Technical Report EPFL Technical...
- Ayvaci, A., Soatto, S., 2009. Motion segmentation with occlusions on the superpixel graph. In: Proc. IEEE International...
- et al.
Efficient graph-based image segmentation
International Journal of Computer Vision
(2004) - Fulkerson, B., Vedaldi, A., Soatto, S., 2009. Class segmentation and object localization with superpixel neighborhoods....
- He, X., Zemel, R., Carreira-Perpindn, M., 2004. Multiscale conditional random fields for image labeling. In: Proc. IEEE...
- et al.
Robust higher order potentials for enforcing label consistency
International Journal of Computer Vision
(2009) - et al.
TurboPixels: fast superpixels using geometric flows
IEEE Transactions on Pattern Analysis and Machine Intelligence
(2009) - Liu, M.Y., Tuzel, O., Ramalingam, S., Chellappa, R., 2011. Entropy rate superpixel segmentation. In: Proc. IEEE...
- et al.
Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features
IEEE Transactions on Medical Imaging
(2012) - et al.
Contour and texture analysis for image segmentation
International Journal of Computer Vision
(2001)
Cited by (41)
Assessment of masticatory performance by geometric measurement of the mixing ability with 2-color chewing gum
2019, Journal of Prosthetic DentistryNovel Arc-Cost Functions and Seed Relevance Estimations for Compact and Accurate Superpixels
2023, Journal of Mathematical Imaging and VisionSuperpixel segmentation of high-resolution remote sensing image based on feature reconstruction method by salient edges
2023, Journal of Applied Remote SensingSuperpixel Segmentation using Symmetric Kernel Function and it's Application in Image Forgery Detection
2023, Proceedings of CONECCT 2023 - 9th International Conference on Electronics, Computing and Communication TechnologiesAdaptive tuning of SLIC parameter K
2021, Multimedia Tools and Applications