Stereo retinex

Abstract

The retinex algorithm for lightness and color constancy is extended to include 3-dimensional spatial information reconstructed from a stereo image. A key aspect of traditional retinex is that, within each color channel, it makes local spatial comparisons of intensity. In particular, intensity ratios are computed between neighboring spatial locations, retinex assumes that a large ratio indicates a change in surface reflectance, not a change in incident illumination; however, this assumption is often violated in 3-dimensional scenes, where an abrupt change in surface orientation can lead to a significant change in illumination. In this paper, retinex is modified to use the 3-dimensional edge information derived from stereo images. The edge map is used so that spatial comparisons are only made between locations lying on approximately the same plane in 3-dimensions. Experiments on real images show this method works well, however, they also reveal that it can lead to isolated regions, which, as a result of being isolated, are incorrectly determined to be grey. To overcome this problem, stereo retinex is extended to allow information that is orthogonal to the space of possible illuminants to propagate across changes in surface orientation. This is accomplished by transforming the original RGB image data into a color space based on coordinates of luminance, illumination and reflectance. This coordinate system allows stereo retinex to propagate reflectance information across changes in surface orientation, while at the same time inhibiting the propagation of potentially invalid illumination information. The stereo retinex algorithm builds upon the multi-resolution implementation of retinex known as McCann99. Experiments on synthetic and real images show that stereo retinex performs significantly better than unmodified McCann99 retinex when evaluated in terms of the accuracy with which correct surface object colors are estimated.

Introduction

Although it is well established that for human subjects that a surface’s perceived spatial location affects the perception of its lightness and color [1], [2], many machine color constancy models [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14] make no use of 3-dimensional spatial information. In fact, many of the methods are based on binarized color histograms, which discard all the images’ spatial structure, and rely instead on statistical properties of the color distribution in order to determine the color of the scene illuminant. Although these methods work quite well [5], they all assume implicitly that there is a single scene illuminant. However, multiple illuminants are common in typical scenes. Outdoors, for example, shadowed areas are not only darker, but much bluer, than those in the sun, because the sky’s light is bluer than the sun’s.

In this paper, we extend retinex to take advantage of 3-dimensional distance information extracted from stereo imagery. In particular, since an abrupt change in surface orientation may lead to an abrupt change in the incident illumination as, for example, occurs due to self-shadowing, retinex is modified so that its computation does not cross edges in the depth map. In this way, it can provide lightness/color estimates for different parts of the scene that may be illuminated differently.

Although this modification of retinex does ameliorate many of problems that arise in mulit-illuminant scenes, the processing has a tendency to result in isolated grey areas. This problem arises especially for surfaces of uniform color that are completely isolated from other surfaces by a change in surface orientation. Retinex normalizes to white, so any completely isolated single color will always be made white (or grey after subsequent intensity adjustment). To overcome this problem, a color coordinate system [27] is used with axes representing variation in illumination color, intensity, and object reflectance. Retinex is applied separately to each of these color channels and the result is then transformed back to the original color coordinates. The coordinate system allows stereo retinex to propagate reflectance information across changes in surface orientation, while at the same time inhibiting the propagation of potentially invalid illumination information.

Tests on synthetic and real images show that the modified, depth-aware stereo retinex method outperforms the original retinex method in terms of the accuracy with which the true scene surface colors are estimated. Accurate estimation of scene colors under uncontrolled illumination conditions is important in many computer vision applications.

Retinex has a long history beginning with an early paper by Land and McCann [15] and there are many variations on the original retinex algorithm. The basic principles of retinex are: (1) color is obtained from 3 ‘lightnesses’ computed separately for each of the color channels; (2) the ratios of intensities from neighboring locations are assumed to be illumination invariant; (3) lightness in a given channel is computed over large regions based on combining evidence from local ratios; (4) the location with the highest lightness in each channel is assumed to have 100% reflectance within that channel’s band. Lightness refers to the perceived (in the case of human perception), or estimated (in the case of computational methods) surface albedo (reflectance averaged over the channel’s band).

The initial versions of retinex where based on combining the ratio information along random paths across the image. Multi-resolution versions of retinex were introduced for efficiency [16]. Horn [17] formalized retinex in terms of differentiation, thresholding and re-integration in the logarithm domain. Kimmel et al. [18] formulate the computation as a variational optimization problem. Two versions of retinex have been given standardized definitions in terms of Matlab code [19].

All of the retinex variants treat the input image as a spatial arrangement of colors and make no use of the 3-dimensional structure of the underlying scene. However, there are a number of psychophysical experiments indicating that the human lightness and color perception are influenced by information from several sources, including 3-dimensional scene geometry. In particular, Gilchrist’s early experiments [20] showed that, in the black and white scenes, changing a surface’s apparent 3-dimensional context affected the perception of its lightness. Gilchrist writes, “The central conclusion of this research is that perceived surface lightness depends on ratios between regions perceived to lie next to one another in the same plane” [20]. The extension to retinex proposed here uses ratios between regions lying next to one another and, furthermore, specifically excludes ratios from neighboring regions lying in different planes. In experiments using computer graphics rendered 3-dimensional scenes, Boyaci et al. [21] provided further evidence for the relationship between perceived orientation and the perceived lightness of matte surfaces. Yamauchi et al. [22] used stereoscopic stimuli to support the notion that surface color perception is strongly influenced by depth information. Bloj et al. [23] illustrated the effect of spatial shape on chromatic recognition. Yang and Shevell [24] show that binocular disparity can improve color constancy. Adelson [25] argues that statistical and spatial arrangement information are combined for lightness perception.

Since there is plenty of psychophysical evidence indicating a connection between a surface’s spatial properties in 3-dimensions and its perceived lightness and color properties, the question is how to include the spatial information into a color constancy model? We investigate how it can be incorporated into the retinex model in particular, and show that spatial information does improve its color constancy performance significantly.