Object-oriented convolutional features for fine-grained image retrieval in large surveillance datasets

Highlights

• Proposed to represent vehicle images with appropriate convolutional features.

• Our method reduces number of feature maps without performance degradation.

• Selected features yield better retrieval performance than the full feature set.

Abstract

Large scale visual surveillance generates huge volumes of data at a rapid pace, giving rise to massive image repositories. Efficient and reliable access to relevant data in these ever growing databases is a highly challenging task due to the complex nature of surveillance objects. Furthermore, inter-class visual similarity between vehicles requires extraction of fine-grained and highly discriminative features. In recent years, features from deep convolutional neural networks (CNN) have exhibited state-of-the-art performance in image retrieval. However, these features have been used without regard to their sensitivity to objects of a particular class. In this paper, we propose an object-oriented feature selection mechanism for deep convolutional features from a pre-trained CNN. Convolutional feature maps from a deep layer are selected based on the analysis of their responses to surveillance objects. The selected features serve to represent semantic features of surveillance objects and their parts with minimal influence of the background, effectively eliminating the need for background removal procedure prior to features extraction. Layer-wise mean activations from the selected features maps form the discriminative descriptor for each object. These object-oriented convolutional features (OOCF) are then projected onto low-dimensional hamming space using locality sensitive hashing approaches. The resulting compact binary hash codes allow efficient retrieval within large scale datasets. Results on five challenging datasets reveal that OOCF achieves better precision and recall than the full feature set for objects with varying backgrounds.

Introduction

In recent years, we have seen tremendous increase in the production and consumption of multimedia data partly due to advent of the social web and partly because of the progress in surveillance, medical, industrial, mobile and embedded computing technologies [1]. Consequently, multimedia data including images and videos are produced and stored in huge amounts. These multimedia repositories contain wealth of highly useful information for administrators and decision makers, provided that efficient and reliable access to relevant data is ensured [2]. Content-based image retrieval (CBIR) systems attempt to locate images containing objects similar to that of a query image by analyzing their contents. CBIR has several applications in information retrieval, surveillance, medical, e-commerce, industry, and social web. Recently, it has attracted a lot of attention due to the rising interest in making the best use of available multimedia data [3]. The exponential increase in the volume of image data, and the inherent complexity of visual contents (projecting 3D world onto a 2D canvas) has made image retrieval increasingly difficult. This difficulty increases even further with fine-grained image retrieval due to the existence of high degree inter-class visual similarity [4]. One such problem arises when retrieving images from traffic surveillance datasets, where the main object of interest are vehicles [[5], [6]]. There exists greater visual similarity despite the fact that vehicles may belong to different categories.

Visual surveillance has become an undeniable necessity of the day, producing huge amounts of multimedia data, which is stored for future analysis [[7], [8]]. Indexing and retrieval of such huge volumes of data requires efficient representation methods [[9], [10]]. Though there exists numerous ways to represent visual contents in large datasets, complexity in the nature of visual data in surveillance limits the use of traditional image representation schemes. Earlier image retrieval methods used local features like scale-invariant features transform (SIFT) [11] and other feature aggregation schemes like vectors of locally aggregated descriptors (VLAD) [12] and fisher vectors (FV) [13]. In recent years, the success of CNN based features prevailed as the state-of-the-art features for image retrieval and classification. Some of the earlier works by Babenko and Lempitsky [14] and Razavian et al. [15] showed that features from a pre-trained CNN can be used to represent images, yielding state-of-the-art performance in large datasets. However, these approaches directly used activations from various layers without considering the suitability of these features for particular object classes.

In this paper, we investigated convolutional features maps of a pre-trained deep CNN to identify a set of optimal features for representing surveillance objects like vehicles for image retrieval applications. A feature selection procedure is presented for vehicles, allowing us to select appropriate features for fine-grained image search. Main contributions of our work are as follows:

• Convolutional activation features have been investigated for vehicles in order to select appropriate features for their effective representation.

• An efficient feature selection procedure is presented through which it is shown that the number of feature maps can be considerably reduced without any degradation in performance. The selected features exhibit greater attention to the object of interest than the background.

• It has also been shown through experiments that the selected features yield better retrieval performance at higher ranks than the full set of features.

The rest of the paper is organized as: Section 2 introduces relevant literature in the field of image retrieval. Section 3 presents schematics of the proposed approach. Experimental results are discussed in Section 4 and the paper is concluded with future research directions in Section 5.