Color CENTRIST: Embedding color information in scene categorization

Highlights

• A color index scheme is designed to embed color information into the framework of CENTRIST.

• Performance of the proposed descriptor is evaluated based on various datasets.

• Extensive performance comparison with original CENTRIST descriptors, local descriptors, and local patterns.

• An application on object detection is proposed to applying color CENTRIST in different domains.

Abstract

A new color descriptor has now been proposed to embed color information into the framework of CENsus TRansform hISTogram (CENTRIST), so that such a state-of-the-art visual descriptor can be further improved to categorize image scenes. In the proposed color CENTRIST descriptor, global structure characteristics are described by both gradients derived from intensity values and color variations between image pixels. The spatial pyramid scheme has also been adopted to convey information in different scales. Comprehensive studies based on various datasets were conducted to verify the effectiveness of the color CENTRIST from different aspects, including the way to quantize the color space, selection of color space, and categorization performance on various datasets. We demonstrated that the color CENTRIST descriptor was not only easy to implement, but also reliably achieved superior performance over CENTRIST. An application was also proposed to demonstrate the possibility of applying the color CENTRIST in various domains.

Introduction

Scene categorization, or scene recognition, has become a fundamental process for efficient image browsing, retrieval, and organization. For example, if an image’s scene category can be recognized, we would reduce the search space of object recognition, or more accurately, detect semantic concepts present in this image. The results of scene categorization may also help a robot to localize itself in a building. Detecting semantic category of an image is undoubtedly important, and devising good visual descriptors plays the core role in this task.

In the literature, many visual descriptors have been proposed for image scene recognition. They can be roughly divided into two groups: (1) part-based representation, with the consideration of multiple scales or spatial distributions, and (2) holistic representation that directly models global configurations. The former approach describes texture information in image patches, and has been proven to be extremely effective when detecting objects under various conditions. By considering the distribution of local descriptors over image patches, sometimes in a multiscale manner, global information is captured. One of the most popular part-based descriptors is Scale-Invariant Feature Transform (SIFT) [7], and one of the most prominent approaches to consider the global distribution is the spatial pyramid approach [3]. Despite the SIFT descriptors associated with the bag of visual words model [8] which have shown discriminative power on scene categorization, directly modeling global texture information often more reliably describes spatial structure of a scene. The same scene may be taken from various viewpoints, and objects with significantly different appearances may appear in the same type of scene. In contrast with the local texture information, holistic representation, such as GIST [2], captures global structure and achieves high accuracy in natural scene categorization. Recently, CENsus TRansform hISTogram (CENTRIST) [1] was proposed to provide accurate and stable performance on various scene image datasets.

We found that most works were targeted on gray images, and existing visual descriptors mainly relied on oriented gradient calculated based on intensity values. However, we argue that color information also plays an important role, although it would not be as important as intensity, and should not be neglected in scene categorization. Fig. 1 shows an example about how color information is used in distinguishing scene categories. Without color information, these two images have similar structure and are hard to recognize. With color information we realize that the open country image has a blue region on the top half, while the coast image has two distinct blue regions at the top half and the bottom half, respectively. It is clear that considering color information benefits scene categorization.

In this work, we devise a visual descriptor called color CENTRIST to embed color information into the framework of CENTRIST, and demonstrate its effectiveness through evaluating various color image datasets. Through comprehensive evaluation, we verify effectiveness of the color CENTRIST. The main contributions of this work are briefly described as follows, which were also shown in our preliminary work [21].

• We devise a color index scheme to embed HSV color information into the framework of CENTRIST. Information of three color channels is encapsulated into an 8-bit representation, so that the framework of CENTRIST can be directly employed, and various performance comparisons can be impartially conducted. We verify that different color channels should be allocated different numbers of bits to more accurately characterize image content.

• Performance of the proposed descriptor is evaluated based on various datasets, including the 8-class scene dataset, the 8-event dataset, the 67-indoor scene dataset, the KTH-IDOL and the KTH-INDECS datasets. Working on various datasets shows robustness and effectiveness of the proposed descriptor.

The unique contributions of this work over our previous work [21] are described as follows.

• We verify the best multilevel representation of the proposed descriptor by carefully evaluating performances obtained by different levels of descriptors. Moreover, statistical analysis is conducted to show that the performance superiority is statistically significant.

• We verify that extracting the proposed descriptor from the HSV color space gives stable performance.

• We verify that combining the proposed descriptor with CENTRIST further yields better performance.

• We compare performance obtained by the proposed descriptor with that obtained by SIFT, and its color variants, based on the bag of words framework.

• We compare performance obtained by the proposed descriptor with that obtained by several promising color LBPs.

• An application on object detection is proposed to demonstrate the possibility of applying the color CENTRIST in different domains.

The rest of this paper is organized as follows. Section 2 provides a literature survey. The color CENTRIST descriptor is proposed after briefly reviewing conventional CENTRIST in Section 3. Preliminary analysis of different descriptor settings is described in Section 4. We provide comprehensive evaluation on various datasets in Section 5, and a novel application based on color CENTRIST in Section 6. Section 7 concludes this paper with discussions of the proposed descriptor and future research.