Making a smarter color camera

Abstract

One meaning of “smart camera” is that it images objects in the class(es) of interest while not imaging objects in other class(es). Such a camera would be most useful if it were software controlled and operated at TV frame rates. In this paper, Artificial Color is applied in the design of such a camera. We show here how to segregate objects by color even when there are nearly identical neighboring colors. The process uses a multiple stage (“divide and conquer”) approach. The method used employs only linear discriminants, so it is simple to implement in software, firmware, or hardware. An example of undoing one of nature’s best camouflage efforts is shown.

Introduction

“Smart cameras capture high-level descriptions of the scene and analyze what they see” [1]. Many things can render a camera smart. For example, smart cameras can be designed for special purposes, e.g. recognizing specific people [2] or recognizing actions [3]. They can also be designed for generalized image analysis. Our work fits in the latter category and works in the spectral domain. An example of prior work in spectrally discriminating smart cameras is the on-focal-plane spectral processing of Chai et al. [4]. The present work fits in that same category as a camera designed to utilize the spectral content of the scene in real time to classify pixels in a scene according to their membership in user-defined classes.

The development spectral imaging technology brings hope for increase in discrimination and identification of objects beyond capabilities of human eye even in the visible spectral region. Human color vision is based on three sensor bands. The spectral sensitivities overlap considerably. In general, Natural Color uses data from two or more overlapping spectral sensitivity detections to compute spectral discriminants that the brain assigns to the object.

Nature solved the problem of broad spectral coverage with good spectral discrimination by sampling not the spectrum but broad, overlapping spectral bands. This provided three major benefits. First, it provided differential spectral information over a large band without using a large number of measurements (as would a spectrometer). For nature, discrimination, not spectral resolution is important. Second, much more light reaches the detector through a broad filter than through a narrow filter, so good sensitivity was achieved. Third, the visual signal processing is a lot simpler and faster if the spectral information is represented by a few numbers rather than by many. Most mammals, for instance, use two such bands. Primates use three.

In human vision, for example, we can say that each portion of the scene is sensed with three types of cone cells in daylight vision. Their spectral sensitivities overlap considerably as shown in Fig. 1. It is the overlap, not the number of types, of sensitivity curves that is the key feature of animal vision. At night, when you have only one type of cell sensitive enough to be useful, a signal detected combines brightness and wavelength information in such a way that they cannot be untangled. You say that you see no color. No single curve can give any spectral contrast or normalization, so no spectral discriminant can be computed.

Artificial Color also computes spectral discriminants using data taken with two or more spectrally overlapping sensitivity curves. The use of a few broad bands rather than many narrow bands, Artificial Color allows cameras to have great sensitivity, size, and cost advantages over hyperspectral imaging while maintaining the high discrimination of hyperspectral cameras.

Our work differs from prior work in two respects.

First, it demonstrates that powerful spectral discrimination is possible using only a few (in this case three) spectrally overlapping filters, e.g. the RGB filters of commercial color cameras. Other works on RGB cameras fail to give the dramatic discrimination our camera will demonstrate here. Hyperspectral cameras will also yield such discrimination, but they have other problems relative to our simple commercial camera, as just suggested. Typical prior work on color discrimination is based on object histogram matching. If the histograms largely overlap as clearly they do with the frog and the dark green leaves, the overlap will be quite substantial and discrimination will be asked to work on small differences in large signals with inevitable variabilities. Antania et al. provide a comprehensive review of such methods [5]. Artificial Color constructs filter that attend only to those spectral feature most useful for discrimination, and (as shown here) can be used in sequence to achieve even better discrimination than any single filter could.

Second, our processing is software based. That is merely a convenience for us, not an inherent speed limitation. The processes we implement amount to nothing more than thresholding linear discriminants in the spectral domain. We showed two decades ago that linear discriminants can be implemented in optical hardware [6], [7] and doing them with current electronic hardware is also easy. On the positive side, a software controlled smart camera is easily reprogrammed to perform new tasks. We illustrate Artificial Color camera tuned for finding green frogs in green foliage. By merely changing the program, it could discriminate between genuine and fake pills, image necrotic areas during cancer operations, or measure the temperature of glowing objects.

The basis for our work is what we have called Artificial Color that will be described next.

This paper not only applies Artificial Color to smart camera design but also shows how it can be used to attack extremely difficult spectral discrimination problems by breaking them into smaller, easier-to-discriminate problems.