FOCUS: A system for searching for multi-colored objects in a diverse image database

Abstract

We describe a new multi-phase, color-based image retrieval system (FOCUS) which is capable of identifying multi-colored query objects in an image in the presence of significant, interfering backgrounds. The query object may occur in arbitrary sizes, orientations, and locations in the database images. Scale and rotation invariant color features have been developed to describe an image, such that the matching process is fast even in the case of complex images. The first phase of processing matches the query object color with the color content of an image computed as the peaks in the color histogram of the image. The second phase matches the spatial relationships between color regions in the image with the query using a spatial proximity graph (SPG) structure designed for the purpose. Processing at coarse granularity is preferred over pixel-level processing to produce simpler graphs, which significantly reduces computation time during matching. The speed of the system and the small storage overhead make it suitable for use in large databases with online user interfaces. Test results with multi-colored query objects from man-made and natural domains show that FOCUS is quite effective in handling interfering backgrounds and large variations in scale. The experimental results on a database of diverse images highlights the capabilities of the system.

Introduction

With the advent of large image databases with complex images, efficient content-based retrieval of images has become an important issue. The aim of content-based retrieval is to find images in a database which contain the object represented in a query image. When the database has images of multi-colored objects which can be recognized on the basis of their distinctive color signatures alone, the color of the object is an obvious choice for indexing. The simpler problem of finding global similarity between a query image and candidate images based on color has been addressed in [9], [13], [22], [32]. However, multi-colored objects usually will not occur by themselves in an image. There may be significant, interfering background clutter in the images or the object may occupy a very small portion of the overall image, making global similarity measures unsuitable for retrieval.

In the general case studied here, the target image may contain the queried object at a scale vastly different from the representation in the query image, at any location and with any orientation. The background in the target image is unconstrained and may include other objects with similar colors. However, it is assumed that the queried object appears in the target images with no occlusion and under similar photometric conditions. Since the proposed method is intended for use with online user interfaces, speed is an important criterion and a response time of a few seconds or less is targeted.

There are many image databases in domains where object color can be a basis for retrieval—flags, logos, consumer products, textile patterns, and postal stamps among man-made objects and flowers, birds, fish, and butterflies as example image databases in the natural domain. The database on which FOCUS has been tested consists of 1200 diverse color images. There are 400 advertisements from magazines and 800 color images from nature including birds, fish, flowers, animals, and vegetables. Advertisements are the most challenging component of the database where the goal is to retrieve all advertisements of a product shown in a query image. This is a particularly complex task since the queried object may appear in candidate images in various sizes and orientations with a wide variety of background colors and forms as shown in Fig. 1. The difference in scale is as large as 1:25 in some query–target pairs in our database. Unlike other databases on which color-based retrieval has been tried (flags, logos, and products), in most of the advertisement images the products do not spatially dominate the image, nor are they necessarily in the center. There is no concept of foreground and background—what is background clutter for this application may actually be the foreground of the image. Consequently, no focus-of-attention pre-segmentation is possible. In the rest of the paper, background refers to all objects and context in the image which are not a part of the queried object.

Our choice of domain gives us some advantages in offsetting the difficulty of the problem. Advertisers want consumers to see their product clearly, so occlusion of the product is rare, and typically the same aspect of the product is presented in all advertisements. There may be small out-of-plane rotations causing some occlusion, but all major colors of the object remain visible. Also, the advertisers take care that their products are printed with their true colors so there is very little color distortion across different advertisements of the same product. However, the color constancy problem still arises because light and shadow effects in images create differences in the apparent color of objects. Fortunately, the color variations are not severe and can be handled by the selection of a robust color representation and by allowing some tolerance in the matching strategy. A brief description of this work appeared in [8].