Content-sensitive superpixel segmentation via self-organization-map neural network

Highlights

• Present a novel metric for the content-sensitiveness of superpixel.

• Proposed a content-sensitive sampling algorithm to generate content-sensitive superpixel.

• Propose a novel superpixel segmentation algorithm based on SOM.

Abstract

Content-sensitive superpixel segmentation generates small superpixels in content-dense regions and large superpixels in content-sparse regions. It achieves higher segmentation accuracy than traditional superpixels. In this paper, we propose a content-sensitive superpixel segmentation algorithm based on Self-Organization-Map (SOM) neural network. First, we propose a novel metric to measure the content-sensitiveness of superpixels. Second, by using this metric, we develop a sampling algorithm to sample pixels from image according to their content-sensitiveness. Finally, a SOM neutral network is trained with the sampled pixels and used to segment the image into content-sensitive superpixels. The Berkeley Image Segmentation database and INRIA database are used to evaluate the proposed method. The experiment results show that the proposed approach outperforms state-of-the-art methods.

Introduction

A superpixel is defined as a set of grouped homogeneous pixels in an image. Superpixel segmentation is also called as oversegmentation. It is different from image segmentation [1], [2] and co-segmentation [3]. Both image segmentation and co-segmentation intend to extract foreground from images, but co-segmentation always segments a series images with similar scene simultaneously. However, superpixel segmentation aims to group pixels into homogeneous regions for the following tasks. Superpixels are used to replace pixels as the atomic unit in the following computer vision tasks. Some image segmentation algorithms can be used to extract superpixels from image, such as QuickShift [4], MeanShift [5] and Normalized cuts [6]. Since it can be used to improve the performance of subsequent computer vision tasks, superpixel segmentation has attracted more and more attentions in recent years and has been widely applied to many applications, such as saliency detection [7], [8], image segmentation [9], [10], image parsing [11], surface reconstruction [12] and object recognition [13], [14].

Many superpixel segmentation methods have been proposed [15], [16], [17], [18], [19], [20], [21], [22], [23], [24]. However, the following challenge of superpixel segmentation is still unsolved: on one hand superpixel segmentation should avoid under-segmentation error and preserve the details of image content; on the other hand, superpixel segmentation requires generating superpixels as few as possible to reduce the computation complexity of the following tasks. This motivates researchers to focus on content-sensitive superpixels segmentation [19], [23].

Content-sensitive superpixel whose size is sensitive to the content density of local regions in the image. The content density of an image region is usually measured by its color variation. It often differs in different parts of the image. Fig. 1 shows an example of such situation. The color variation of pixels in the green window in Fig. 1 is smaller than that in the red window. Traditional superpixel segmentation algorithms ignore such difference. The superpixels they generated usually have similar size and shape. On the other hand, content-sensitive approaches can produce small superpixels in regions have high color variation and large superpixels in regions have low color variation. As pointed out by Liu [19], this is a better image representation than traditional uniform superpixel, which has lower under-segmentation error and can preserve more details of image boundary.

The previous methods of content-sensitive superpixel segmentation include SSS [23] and Manifold SLIC [19]. In the SSS, two steps are used to generate superpixels. First, the centers of each superpixel are roughly placed in a lattice structure on the image. Then, the centers are relocated or split repeatedly until the energy function which is defined on geodesic distances between pixels and the compactness of superpixels meets the termination condition. The energy function consists of two terms. The first term embeds the color homogeneity of pixels and the second item integrates the content density and compactness constraints. The content-sensitiveness of superpixel can be adjusted using the balance factor between the two terms. Manifold SLIC extends traditional superpixel algorithm of Simple Linear Iterative Clustering (SLIC) to generate content-sensitive superpixels by mapping the image into a 2-dimensional manifold, in which the content density of the image is measured. Actually, it uses an efficient algorithm to compute restricted centroidal Voronoi tessellation on the manifold. Then the content-sensitiveness of superpixels is measured by the areas of Voronoi cells on the 2-dimensional manifold after mapping and the content-sensitive superpixels are produced accordingly. Manifold SLIC runs 10 times faster than SSS. Although the two methods above try to compute the content-sensitive superpixels, the explicit measurement of content-sensitiveness has not been explored. In this paper, we introduce a metric to measure the content-sensitiveness of superpixels, based on which we propose a superpixel segmentation method by using Self-Organization Map (SOM) neural network. We call the proposed method SOMS (SOM Superpixel) for short. Actually, we develop a content-sensitive sampling method to get pixels from the image. Then these pixels are used to train a SOM neural network for clustering pixels into content-sensitive superpixels. The main contributions of this paper are summarized as follows:

• A novel metric to measure the content-sensitiveness of superpixels is proposed.

• With this novel metric, we put forward a content-sensitive sampling algorithm to get pixels from images.

• We present the SOMS algorithm by training a SOM on the sampled pixels and use it to cluster image into superpixels.

This paper is organized as follows. In Section 2, we describe our SOMS algorithm. In Section 3, our approach is thoroughly evaluated and compared with other state-of-the-art approaches. Section 4 concludes this paper.