Elsevier

Pattern Recognition Letters

Volume 43, 1 July 2014, Pages 71-80
Pattern Recognition Letters

An evaluation of the compactness of superpixels

https://doi.org/10.1016/j.patrec.2013.09.013Get rights and content

Highlights

  • We propose a metric to measure the compactness of superpixel segmentations.

  • We show that there is a negative correlation between compactness and boundary recall.

  • The proposed algorithm offers a transparent compactness control.

  • The segmentation optionally conforms to a superpixel lattice.

  • An example application demonstrates the relevance of controlling compactness.

Abstract

Superpixel segmentation is the oversegmentation of an image into a connected set of homogeneous regions. Depending on the algorithm, superpixels have specific properties. One property that almost all authors claim for their superpixels is compactness. However, the compactness of superpixels has not yet been measured and the implications of compactness have not been investigated for superpixels. As our first contribution, we propose a metric to measure the compactness of superpixels. We further discuss implications of compactness and demonstrate the benefits of compact superpixels with an example application. Most importantly, we show that there is a negative correlation between compactness and boundary recall. A second desirable property for superpixel segmentations is conforming to a lattice. This regular structure, similar to the pixel grid of an image, can then be used for more efficient algorithms. As our second contribution, we propose an algorithm that offers both a transparent and easy-to-use compactness control with an optional lattice guarantee. We show in our evaluation with six benchmark algorithms, that the proposed algorithm outperforms the state-of-the-art.

Introduction

Image segmentation is a fundamental task in computer vision and many applications rely on it as a preprocessing step. Superpixel segmentation belongs to the class of oversegmentation algorithms and the term superpixel was introduced by Ren and Malik (2003).

A superpixel is defined as a homogeneous image region that aligns well with object boundaries. This allows to represent an image with only a couple of hundred segments instead of tens of thousands of pixels. This reduction of input complexity makes superpixels particularly useful for a wide range of application domains, for example image segmentation (Achanta et al., 2010, Schick and Stiefelhagen, 2011, Veksler et al., 2010), object recognition (Achanta et al., 2010), object localization (Fulkerson et al., 2009), labeling tasks (Kohli et al., 2009), motion segmentation (Ayvaci and Soatto, 2009), foreground segmentation (Schick et al., 2012), tracking (Wang et al., 2011), and pose estimation (Mori, 2005, Mori et al., 2004), to name just a few.

A compact superpixel has a regular shape with smooth boundaries and many authors agree that compactness is desirable for superpixels (Achanta et al., 2010, Levinshtein et al., 2009, Liu et al., 2011, Moore et al., 2010, Veksler et al., 2010, Zeng et al., 2011, Zhang et al., 2011, Perbet and Maki, 2011). However, the compactness of superpixel segmentations has not yet been systematically measured and evaluated. We are the first to measure the compactness of superpixels and investigate its implications.

This work is an extended version of Schick et al. (2012). The main additional contribution is an extension of the superpixel segmentation in Schick et al. (2012) that guarantees that it conforms to a lattice. Further, we extended the experimental section with more discussions about compactness including a correlation, convergence, and lattice stability analysis.

The remainder of this paper is organized as follows. Related work is discussed in Section 2. The compactness metric is presented in Section 3. Section 4 introduces the segmentation algorithm followed by the proposed lattice constraints in Section 5. The evaluation is presented in Section 6 with results in Section 7. Section 8 demonstrates the benefits of compactness with an example application before the conclusion in Section 9.

Section snippets

Related work

Superpixels have received increasing attention in the last years and there is a wide range of superpixel segmentation algorithms. These algorithms differ in how they solve the segmentation task which results in different properties regarding runtime, segmentation quality, and superpixel shape. Graph-based segmentation algorithms were proposed by Shi and Malik (2000), a normalized cut approach by Malik et al. (2001), and a graph cut approach by Veksler et al. (2010). Zhang et al. (2011) proposed

Superpixel compactness

The larger the area of a shape for a given boundary length, the higher is its compactness. The same holds for superpixels and there seems to be an intuitive understanding that compactness is indeed a desirable property (Achanta et al., 2010, Levinshtein et al., 2009, Liu et al., 2011, Moore et al., 2010, Veksler et al., 2010, Zeng et al., 2011, Zhang et al., 2011, Perbet and Maki, 2011). We will now discuss why compactness is indeed advantageous before explaining the compactness metric.

Superpixel segmentation

In this section, we propose a modification of SLIC (Achanta et al., 2010) that computes more accurate superpixels with a transparent control of their compactness. (This algorithm was also presented in Schick et al. (2012).) SLIC is based on an iterative k-means clustering and computes superpixels utilizing both a distance in color space as well as Euclidean space. While the k-means algorithm leads to very accurate clusters, it does not guarantee that the clusters remain connected which is

Superpixel lattice

In this section, we present a modification of the algorithm proposed in Section 4 that additionally guarantees that the superpixels conform to a regular lattice. We first discuss the advantages of superpixel lattices before introducing the lattice constraints.

Evaluation

We compared the proposed algorithm to five different superpixels algorithms. The superpixels algorithms are the normalized cuts segmentation from Mori, 2005, Mori et al., 2004 (NC), SLIC (Achanta et al., 2010), TurboPixels (Levinshtein et al., 2009) (TP), entropy rate superpixels (Liu et al., 2011) (ERS), and Superpixel lattices (Moore et al., 2008) (LATTICE). SLIC allows to weight its Euclidean distance term which we included in the evaluation by testing the extreme values (SLIC_min, SLIC_max)

Results

This section presents the evaluation results. We first discuss quantitative results of all four metrics before presenting our compactness-specific findings. After discussing the impact of the lattice constraints on performance, we conclude with a set of images for a more qualitative impression.

Application

We showed in Section 1 that superpixels are applied to a wide range of different applications. Which superpixel properties are most important depends on the application in question. For some applications, the shape might not be important, e.g. when computing mean color values or histograms. For applications, however, that work directly with superpixel boundaries or that require a regular shape (Mori, 2005, Mori et al., 2004), the superpixel compactness might be very important. We will now

Conclusion

Compactness is an important and desirable property of superpixels (Achanta et al., 2010, Levinshtein et al., 2009, Liu et al., 2011, Moore et al., 2010, Veksler et al., 2010, Zeng et al., 2011, Zhang et al., 2011, Perbet and Maki, 2011). With this work, we proposed a metric to measure the compactness of superpixel segmentations. We further discussed implications and showed that there is a negative correlation between boundary recall and compactness. The advantages of controlling the compactness

Acknowledgments

This work was partially supported by the FhG Internal Programs under Grant No. 692026 and by OSEO, French State agency for innovation, as part of the Quaero Programme.

References (32)

  • Achanta, R., Smith, K., Lucchi, A., Fua, P., Ssstrunk, S., 2010. SLIC Superpixels. Technical Report EPFL Technical...
  • Ayvaci, A., Soatto, S., 2009. Motion segmentation with occlusions on the superpixel graph. In: Proc. IEEE International...
  • P.F. Felzenszwalb et al.

    Efficient graph-based image segmentation

    International Journal of Computer Vision

    (2004)
  • Fulkerson, B., Vedaldi, A., Soatto, S., 2009. Class segmentation and object localization with superpixel neighborhoods....
  • He, X., Zemel, R., Carreira-Perpindn, M., 2004. Multiscale conditional random fields for image labeling. In: Proc. IEEE...
  • P. Kohli et al.

    Robust higher order potentials for enforcing label consistency

    International Journal of Computer Vision

    (2009)
  • A. Levinshtein et al.

    TurboPixels: fast superpixels using geometric flows

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2009)
  • Liu, M.Y., Tuzel, O., Ramalingam, S., Chellappa, R., 2011. Entropy rate superpixel segmentation. In: Proc. IEEE...
  • A. Lucchi et al.

    Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features

    IEEE Transactions on Medical Imaging

    (2012)
  • J. Malik et al.

    Contour and texture analysis for image segmentation

    International Journal of Computer Vision

    (2001)
  • Martin, D., Fowlkes, C., Tal, D., Malik, J., 2001. A database of human segmented natural images and its application to...
  • Moore, A., Prince, S., Warrell, J., Mohammed, U., Jones, G., 2008. Superpixel lattices. In: Proc. IEEE Conference on...
  • Moore, A., Prince, S.J.D., Warrell, J., 2010. Lattice cut – constructing superpixels using layer constraints. In: Proc....
  • Mori, G., 2005. Guiding model search using segmentation. In: Proc. IEEE International Conference on Computer Vision,...
  • Mori, G., Ren, X., Efros, A., Malik, J., 2004. Recovering human body configurations: combining segmentation and...
  • Nowozin, S., Gehler, P.V., Lampert, C.H., 2010. On parameter learning in CRF-based approaches to object class image...
  • Cited by (41)

    View all citing articles on Scopus
    View full text