Unsupervised image segmentation based on analysis of binary partition tree for salient object extraction

doi:10.1016/j.sigpro.2010.07.006

Signal Processing

Volume 91, Issue 2, February 2011, Pages 290-299

https://doi.org/10.1016/j.sigpro.2010.07.006 Get rights and content

Abstract

This paper proposes an unsupervised image segmentation approach aimed at salient object extraction. Starting from an over-segmentation result of a color image, region merging is performed using a novel dissimilarity measure considering the impact of color difference, area factor and adjacency degree, and a binary partition tree (BPT) is generated to record the whole merging sequence. Then based on a systematic analysis of the evaluated BPT, an appropriate subset of nodes is selected from the BPT to represent a meaningful segmentation result with a small number of segmented regions. Experimental results demonstrate that the proposed approach can obtain a better segmentation performance from the perspective of salient object extraction.

Introduction

Salient object extraction from images and videos is usually an important part in many multimedia applications such as object-based coding, object-based image/video retrieval, image/video editing and manipulation, smart video surveillance, and human computer interaction. As a preceding step, a suitable image segmentation result will greatly facilitate the following process of salient object extraction. In view of the requirement for salient object extraction, the most preferred image segmentation result should be represented by possibly fewer segmented regions that can still preserve the boundaries of salient objects well. A number of traditional image segmentation approaches such as [1], [2], [3], [4], [5], [6] are generally exploited to obtain a region segmentation result, in which pixels in each segmented region share similar intensity, color or texture, but the problems of over-segmentation and under-segmentation are usually unavoidable. Furthermore, it is possible to obtain a hierarchical segmentation result with different number of segmented regions by directly adjusting some parameters in the so-called multi-scale (multi-resolution) segmentation approaches [7], [8], [9], but it requires a set of manually tuned parameters to obtain a suitable segmentation result for salient object extraction.

Binary partition tree (BPT) was introduced in [10] to systematically represent the hierarchical segmentation of an image in an efficient way. Starting from an initial over-segmentation result generated by any image segmentation approach, the simple yet effective region merging scheme can be exploited to progressively merge adjacent regions based on some kind of dissimilarity measures. The merging sequence can be efficiently recorded by BPT, in which each leaf node represents each initially segmented region and each non-leaf node represents the newly generated region during the region merging process. By using nodes at different levels in BPT to represent the image, it is convenient to obtain a segmentation result at any scale (with any number of segmented regions). By incorporating prior knowledge of a specific class of objects, automatic extraction of face objects [11], [12] and moving objects [13] can be realized by analysis on individual node in the BPT. For general salient object extraction without any prior knowledge, BPT analysis is also useful for highlighting salient regions in the image. In previous works, BPT simplification based on evolvement of region statistics is proposed for convenient tree visualization [14], and is used for efficient image segmentation and interactive extraction of salient objects [15].

In this paper, we present a novel BPT analysis work for unsupervised image segmentation, which shows the suitability for the application of salient object extraction. From an over-segmentation result and the generated BPT, the proposed BPT analysis algorithm automatically selects an appropriate subset of nodes to represent a more meaningful segmentation result. Compared with the previous works [14], [15] based on BPT analysis, the main contribution of our work is twofold. One is that a novel dissimilarity measure considering the impact of color difference, area factor and adjacency degree in a unified way is proposed for region merging and used in the BPT generation process. The other is the proposed BPT analysis algorithm, in which the node evaluation is designed to reasonably identify salient regions, and the following two-phase node selecting process guarantees a meaningful segmentation result possibly reserving salient regions. An obvious feature of our approach is totally free of threshold, while the previous works [14], [15] need user-supplied thresholds during the BPT analysis process. As an unsupervised image segmentation approach, our approach improves the segmentation performance from the view of salient object extraction.

The remainder of this paper is organized as follows. Section 2 describes the process of BPT generation with region merging from an initial segmentation. Section 3 details the proposed BPT analysis algorithm for a meaningful segmentation. Experimental results are shown in Section 4, and conclusion is given in Section 5.

Section snippets

BPT generation from initial segmentation

The original image can be initially partitioned into a set of homogenous regions using a collection of existing image segmentation approaches. The only issue when using any image segmentation approach and possibly adjusting its parameters is to avoid under-segmentation. In other words, the only requirement for initial segmentation is that each segmented region should possibly not cover the parts from different salient objects and background. In this paper, watershed transform [1] is exploited

BPT analysis for meaningful segmentation

In this section, we propose a systematic BPT analysis algorithm to select a suitable subset of nodes to represent a more meaningful segmentation. The proposed algorithm consists of the following two stages, that is, BPT node evaluation and BPT node selection, which are detailed in the following two subsections, respectively.

Experimental results

In order to evaluate the performance of the proposed BPT analysis based unsupervised image segmentation approach from the view of salient object extraction, we select 160 images containing at least one obvious salient object from Berkeley segmentation dataset (BSD) [16], Corel photo gallery, and our image collection. Experimental results on four representative test images are shown in Fig. 3, in which original images, initial segmentation results, final segmentation results generated using our

Conclusion

In this paper, we have presented an efficient unsupervised image segmentation approach based on the BPT analysis. From an initial segmentation result of the original image, a BPT is generated with the region merging process, which is controlled by a novel dissimilarity measure considering the impact of color difference, area factor, and adjacency degree in a unified way. By a systematic analysis of the evaluated BPT, a more meaningful segmentation result is represented by a small subset of

Acknowledgments

The authors are grateful to the anonymous reviewers and the handling editor for their valuable comments, which have greatly helped us to make improvements. This work is supported by National Natural Science Foundation of China under Grant No. 60602012, Shanghai Educational Development Foundation under Grant No. 2007CG53, and Innovation Program of Shanghai Municipal Education Commission (No. 09YZ02).

References (17)

X. Yang et al.
Image segmentation with a fuzzy clustering algorithm based on ant-tree
Signal Process.
(2008)
F.A. Tab et al.
Scalable multiresolution color image segmentation
Signal Process.
(2006)
F. Marques et al.
Face segmentation and tracking based on connected operators and partition projection
Pattern Recognition
(2002)
Z. Liu et al.
An efficient face segmentation algorithm based on binary partition tree
Signal Process.: Image Commun.
(2005)
L. Vincent et al.
Watersheds in digital spaces: an efficient algorithm based on immersion simulations
IEEE Trans. Pattern Anal. Machine Intell.
(1991)
K. Haris et al.
Hybrid image segmentation using watersheds and fast region merging
IEEE Trans. Image Process.
(1998)
R. Nock et al.
Statistical region merging
IEEE Trans. Pattern Anal. Machine Intell.
(2004)
J. Shi et al.
Normalized cuts and image segmentation
IEEE Trans. Pattern Anal. Machine Intell.
(2000)

There are more references available in the full text version of this article.

Cited by (22)

Content-aware image resizing: An improved and shadow-preserving seam carving method
2019, Signal Processing
Citation Excerpt :
Saliency detection approaches are categorized into two groups: bottom-up methods and top-down methods. The bottom-up methods are based on low-level features such as color, intensity, and orientation [26–30]; while top-down methods employ semantic information such as face and text [31]. In the proposed method inspired by the idea given in [32], for extracting the co-saliency map from the multiple images, we introduce a simple yet efficient bottom-up saliency detection method.
The performance of seam carving-based image resizing algorithms is strongly dependent on the quality of the importance map extracted from the image. To date, various approaches have been proposed to extract the importance map, however, none has considered to take the shadows within the image into account. In most cases, existing shadows in the images would imply important information and help in better and quick understanding of the content of the images. This fact motivates us to keep them during the image resizing as much as we can. Therefore, in this paper, a seam carving-based algorithm is presented where we introduce an efficient shadow extraction algorithm in the direction of our motivation. We also propose a saliency map for highlighting the salient objects in the images. In addition to these two maps, a gradient map is also extracted to portray the details of the background. These maps are then combined in order to produce the final importance map. Extensive experiments conducted on a large collection of images indicate that the proposed method, in terms of preserving the main content of the input image, the shadows within it, and the important structures of the image, is superior to state-of-the-art algorithms.
Towards real-time crops surveillance for disease classification: exploiting parallelism in computer vision
2017, Computers and Electrical Engineering
Considering the incessantly increasing economic losses due to plant diseases in the agricultural sector, we have designed a real-time system capable of classifying plant diseases. In this context, we have proposed an image processing algorithm that transforms the image into three colorspaces, which are processed simultaneously. The algorithm executes in a series of intermediate steps, including contrast stretching, feature vector construction, and identification of salient regions. To enable effective execution, we have also proposed the underlying On-Chip communication architecture that allows efficient interconnection between the three digital signal processing cores, each processing its own colorspace. The architecture has been synthesized for 90 nm process, as well as on an FPGA, achieving a post-layout operational frequency of 644 MHz, and an area of 1208.9 µm² on the die. We demonstrate that our system outperforms few existing works in literature in terms of accuracy and computation time.
A fast region segmentation algorithm on compressed gray images using Non-symmetry and Anti-packing Model and Extended Shading representation
2016, Journal of Visual Communication and Image Representation
Citation Excerpt :
An efficient representation method can not only save the storage space of images, but also can reduce the time required for some image manipulations [1–3].
Image segmentation is one of the fundamental steps in image analysis for object identification. The main goal of image segmentation is to recognize homogeneous regions within an image as distinct and belonging to different objects. Inspired by the idea of the packing problem, in this paper, we propose a fast $O (N α (N))$ -time algorithm for image segmentation by using Non-symmetry and Anti-packing Model and Extended Shading representation, which was called the NAMES-based algorithm, where N is the number of homogenous blocks and $α$ ( $N)$ is the inverse of the Ackerman’s function and it is a very slowly growing function. We first put forward four extended Lemmas and two extended Theorems. Then, we present a new scanning method used to process each NAMES block. Finally, we propose a novel NAMES-based data structure used to merge two regions. With the same experimental conditions and the same time complexity, our proposed NAMES-based algorithm, which extends the popular hierarchical representation model to a new non-hierarchical representation model, has about 86.75% and 89.47% average execution time improvement ratio when compared to the Binary Partition Tree (BPT)-based algorithm and the Quadtree Shading (QS)-based algorithm which has about 55.4% execution time improvement ratio when the QS-based algorithm itself is compared to the previous fastest region segmentation algorithm by Fiorio and Gustedt whose $O (N^{2})$ -time algorithm is run on the original $N \times N$ gray image. Further, the NAMES can improve the memory-saving by 28.85% (5.04%) and simultaneously reduce the number of the homogeneous blocks by 49.05% (36.04%) more than the QS (the BPT) whereas maintaining the satisfactory image quality. Therefore, by comparing our NAMES-based algorithm with the QS-based algorithm and the BPT-based algorithm, the experimental results presented in this paper show that the former has not only higher compression ratio and less number of homogenous blocks than the latter whereas maintaining the satisfactory image quality, but also can significantly improve the execution speed for image segmentation, and therefore it is a much more effective algorithm for image segmentation.
Use of Binary Partition Tree and energy minimization for object-based classification of urban land cover
2015, ISPRS Journal of Photogrammetry and Remote Sensing
Two main challenges are faced when classifying urban land cover from very high resolution satellite images: obtaining an optimal image segmentation and distinguishing buildings from other man-made objects. For optimal segmentation, this work proposes a hierarchical representation of an image by means of a Binary Partition Tree (BPT) and an unsupervised evaluation of image segmentations by energy minimization. For building extraction, we apply fuzzy sets to create a fuzzy landscape of shadows which in turn involves a two-step procedure. The first step is a preliminarily image classification at a fine segmentation level to generate vegetation and shadow information. The second step models the directional relationship between building and shadow objects to extract building information at the optimal segmentation level. We conducted the experiments on two datasets of Pléiades images from Wuhan City, China. To demonstrate its performance, the proposed classification is compared at the optimal segmentation level with Maximum Likelihood Classification and Support Vector Machine classification. The results show that the proposed classification produced the highest overall accuracies and kappa coefficients, and the smallest over-classification and under-classification geometric errors. We conclude first that integrating BPT with energy minimization offers an effective means for image segmentation. Second, we conclude that the directional relationship between building and shadow objects represented by a fuzzy landscape is important for building extraction.
Data field-based transition region extraction and thresholding
2012, Optics and Lasers in Engineering
Thresholding is a popular image segmentation method that converts a gray level image into a binary image. In this paper, we propose a data field-based method for transition region extraction and thresholding, which involves three major steps, including generating the image data field, deriving the transition region by comparing the potential values, and calculating the threshold from the transition region. Image data field can effectively represent the spatial interactions of neighborhood pixels, and its potential value is a more robust measurement for the gray level change. In addition, we introduce a fully automatic scheme for parameters selection. The approach is validated both quantitatively and qualitatively. Compared with existing relative methods on a variety of synthetic and real images, with or without noisy, the experimental results suggest that the presented method is efficient and effective.
Saliency-directed color image segmentation using modified particle swarm optimization
2012, Signal Processing
Citation Excerpt :
The watershed algorithm [14], a typical region-based segmentation technique, often produces a lot of small but homogeneous regions, which need some merging operations to reduce the number of regions [15,16]. Liu et al. [16] proposed an unsupervised image segmentation approach. Starting from an over-segmentation color image, region merging is performed using a dissimilarity measure considering color difference, area factor, and adjacency degree, and a binary partition tree (BPT) is generated to record the whole region-merging sequence.
Color image segmentation, an ill-posed problem, can be treated as a process of dividing a color image into some constituent regions and each region is homogeneous. In this study, a saliency-directed color image segmentation approach using “simple” modified particle swarm optimization (PSO) is proposed, in which both low-level features and high-level image semantics extracted from each color image are employed. To extract high-level image semantics from each color image, the visual attention saliency map for each color image is generated by three (color, intensity, and orientation) feature maps, which is used to guide region merging using “simple” modified PSO and a hybrid fitness function for color image segmentation. The proposed approach contains four stages, namely, color quantization, feature extraction, small region elimination, and region merging using “simple” modified PSO. Based on the experimental results obtained in this study, as compared with four comparison approaches, the proposed approach usually provides the better color image segmentation results.

View all citing articles on Scopus

View full text

Unsupervised image segmentation based on analysis of binary partition tree for salient object extraction

Abstract

Introduction

Section snippets

BPT generation from initial segmentation

BPT analysis for meaningful segmentation

Experimental results

Conclusion

Acknowledgments

Signal Process.

Signal Process.

Pattern Recognition

Signal Process.: Image Commun.

Watersheds in digital spaces: an efficient algorithm based on immersion simulations

IEEE Trans. Pattern Anal. Machine Intell.

Hybrid image segmentation using watersheds and fast region merging

IEEE Trans. Image Process.

Statistical region merging

IEEE Trans. Pattern Anal. Machine Intell.

Normalized cuts and image segmentation

IEEE Trans. Pattern Anal. Machine Intell.