Compositing images through light source detection

Abstract

Compositing an image of an object into another image is a frequently occurring task in both image processing and augmented reality. To ensure a seamless composition, it is often necessary to infer the light conditions of the image to adjust the illumination of the inserted object. Here, we present a novel algorithm for multiple light detection that leverages the limitations of the human visual system (HVS) described in the literature and measured by our own psychophysical study. Finally, we show an application of our method to both image compositing and synthetic object insertion.

Introduction

This paper deals with the problem of obtaining the positions and relative intensities of light sources in a scene, given only a photograph as input. This is generally a difficult and under-constrained problem, even if only a single light source illuminates the depicted environment.

Traditionally, light probes are used to acquire the lighting data [1], [2]. A light probe is an object of known 3D shape and BRDF properties (Bidirectional Reflectance Distribution Function, which is a description of the reflectance properties of the material) that is positioned in the scene when the image is captured. Unfortunately, in several cases, this technique is not applicable; e.g., in paintings and in photographs taken under uncontrolled conditions. It would be possible to use any object in the image if geometry information was available to allow light source positions or directions to be estimated [3], [4]. Conversely, if the light source is known, the 3D geometry can be approximately recovered, an ill-posed problem known as shape-from-shading [5].

However, we are interested in the problem of light source recovery without the benefit of any geometric prior models. To this end, we first carried out a psychophysical experiment to quantify the accuracy with which humans can generally detect light sources. The results of this experiment were then used to validate the results of our light-detection algorithm, both numerically and perceptually. We then used any existing object in the image as a de facto light probe. We found that assuming a globally convex shape for such a light probe is sufficient to reconstruct light directions. The user only needs to identify the silhouette of the object in the image, a task similar to or simpler than other existing image-editing applications [6], [7]. We then analyzed the information in the contour and the gradients contained in the shape to infer the light directions and relative intensities.

Real environments are likely to contain multiple light sources. In practice, we found that identifying up to four sources that when combined provided similar illumination as in the image sufficed for most situations. This keeps the dimensionality of the solution manageable, in a way similar to professionally lit environments, which are usually lit by a three-light setup. Additionally, although we did assume in principle that the chosen light probe was Lambertian, we will show that this is not a strong requirement.

We believe that by analyzing the lighting consistency between images, our algorithm can help improve several types of applications, such as Photo Clip Art [7], Interactive Digital Photomontage [8] or Photo Tourism [9].