A unified framework for scene illuminant estimation

Abstract

Most existing illuminant estimation algorithms work with assumptions of specific source types (e.g. directional light source or point light source). The assumptions bring up two main limitations which significantly restrict their applicabilities: first, the prior knowledge about source types is needed; second, it can not handle complex scenes where multiple source types co-exist. In this paper, we develop a general light source model which is designed to model arbitrary light sources. With the general source model, we are able to estimate multiple illuminants of different types within a single unified framework. We use a plastic sphere to probe both diffuse and specular reflections of a scene. With specular reflections, we estimate geometric parameters of the illuminants by a novel ray tracing and matching algorithm. The diffuse reflections are used to estimate photometric parameters of the illuminants. We also notice that common source types are degenerate cases of the general source model under certain conditions. Therefore, we can approximate light sources with common source types by checking the conditions. The approximations serve as type determinations of light sources, which makes our framework extremely useful for lighting related algorithms which need prior knowledge about light sources. Experiment results on a variety of real images demonstrate the efficiency and accuracy of our algorithm.

Introduction

Extensive light source models have been used in computer vision and computer graphics, among which the most common ones are point light source, directional light source and area light source. Each of these models provide an appropriate approximation to a specific category of the real light sources. In real applications, the situation could be very complex where multiple different types of light sources co-exist in the scene. Unfortunately, most light source estimation algorithms work with the assumption of one specific type of light source and there presently exists no method in computer vision for inferring the types of light sources from images. This is problematic because choosing which model to apply in a lighting related problem requires the prior knowledge about the types of the light sources presented in the scene. It is desirable to have a general light source model which allows us to estimate the light sources of different types within a single framework.

Most previous approaches for illuminant estimation make the assumption of directional light source. For example, many schemes have been proposed in the context of shape from shading [23], [1], [10], [6], [21] to estimate a single directional light source. Recently, many attempts have been made to detect multiple directional light sources by exploring various cues. Sato and Ikeuchi [15] estimated illumination distribution by analyzing the shadows cast by an object of known shape onto another object in the scene. Zhou and Kambhamettu [24] utilized the specular properties of a shining surface to estimate the light source directions; Yang and Yuille [20] introduced the idea of occluding boundary to constrain the light source directions; Zhang and Yang [22] extended the idea of occluding boundary and estimate lighting direction from critical points that have surface normals perpendicular to an illuminant direction. Since the detection of critical points is sensitive to noise, Wang and Samaras [18], [19] further segmented the surface into “virtual light patches” by the critical boundaries and illuminants are estimated by minimizing a global error. Li et al. [9] integrated the shading, shadow and specular cues into a single framework which results in more robust estimation.

All the above methods assume directional light sources, which are reasonable approximations to real scene illuminants provided that the distances between the light sources and the illuminated objects are large enough compared against the sizes of both the light sources and the illuminated objects. However, this approximation may not be valid in some cases, especially when light sources are within a limited range, e.g. a small room. The proximal point source model could be used in this situation such as in [12], [8], which estimates the 3-D positions instead of the directions of the illuminants. Nevertheless, all those algorithms are designed for one specific type of the light source which require prior knowledge about the types of the light sources presented in the scene.

To estimate the complex scene illuminations without type assumption, Debevec [2] introduced a light-based model. The model is constructed by mapping reflections on a spherical mirror onto a geometric model of the scene. Although the algorithm works well on superimposing virtual objects onto the real image with convincing shadings, the geometries of all objects in the local scene have to be given a priori, while shape modeling of real objects itself is a difficult research topic. Sato el al. [14] addressed the problem with a radiance-map model. They first build a rough geometric model of the scene as a triangular mesh from a pair of omni-directional images taken from different locations, then the radiance map is constructed by mapping the radiance of the scene onto the reconstructed geometrical model. Although these models have been successfully utilized to render virtual object onto the real images, they were designed specifically for the Augmented Reality and consequently can not be used as a general light source model for all lighting related applications in both computer vision and computer graphics. Specifically, these models suffer from three limitations: first, the direct and indirect illuminations are not explicitly distinguished, which is problematic when the applications only need the information about the direct illuminations (as is the case for most computer vision algorithms); second, the illuminations are estimated as a radiance map while the parameters of the light sources such as the distance and the size of the light source are not explicitly estimated; finally, the geometric model of the scene should be built a priori or be roughly estimated as a triangular mesh, therefore, the estimations of the light sources are expensive. These limitations may not be problems for the applications of Augmented Reality, while for most computer vision algorithms, such as shape from shading and photometric stereo, accurate estimation of the parameters of the light sources such as the directions and the distances of the light sources (direct illuminations) is crucial for the success of the algorithms. To our knowledge, the most closely related work is presented recently by Takai et al. [17] which estimated the point light sources and the directional light sources in a single framework by analyzing the intensity difference of two diffuse spheres although the case of area light source is not considered.

In [5], Langer et al. demonstrated that the set of rays within a free space is a 4-D manifold. Modeling the light sources in terms of the emitting source rays, various light sources can be classified as light sources with different dimensions ranging from 0 to 4. Therefore, the analysis of all different types of light sources could be performed within a 4-D light source hypercube. In this paper, we extend the idea of the 4-D hypercube and propose a general light source model which is more suitable for the purpose of the light source estimation. With this general light source model, different types of light sources can be treated in the same manner. In Section 2, we discuss the light source modeling in detail. The purpose of this work is to design a unified light source estimation framework. Employing the general light source model, we remove the assumption about the source types, therefore, it works on all different types of light sources and consequently can deal with complex scenes where different types of light sources co-exist.

Checking the relationship between the general light source model and the common source types, we notice that the common source types are actually the degenerate cases of the general source models which satisfy certain conditions. Checking these conditions, we can approximate the light sources with the most appropriate common source types. The approximation serves as the type determination of the light sources presented in the scene, which makes our algorithm extremely useful for most lighting related algorithms which need the prior knowledge about the source types.

The rest of this paper is organized as follows: in Section 2, we propose a general light source model for all different types of light sources. Section 3 describes the unified estimation framework in detail. In Section 4, we discuss the conditions upon which the appropriate type approximation can be made to the light sources. Section 5 shows the experiment results on real images and Section 6 gives conclusions and future works.