GPU-friendly rendering for illumination adjustable images

Abstract

An illumination adjustable image (IAI) contains a large number of pre-recorded reference images under various lighting directions. It describes the appearance of a scene illuminated under various lighting directions. Synthesized images about the scene illuminated by complicated lighting conditions are generated from those reference images. This paper presents practical and real-time rendering methods for IAIs based on spherical Gaussian kernel functions (SGKFs). The lighting property of an IAI is represented by a few number of lightmaps. With those lightmaps, we can use consumer-level graphics processing units (GPUs) to perform the rendering process. The rendering methods for directional light source, point light source and slide projector are discussed. Compared with the conventional spherical harmonic (SH) approach, the proposed SGKF approach offers similar distortion performance but it consumes less graphics memory and has a faster rendering speed.

Introduction

Lighting is an essential component in computer graphics and visualization [18], [19]. With a good lighting effect, the content and variation of a scene are enriched. However, geometric-based methods are difficult to instantly render a scene with complex lighting effects. The bottleneck comes from the scene complexity and intensive lighting computation.

Image-based relighting (IBR) [32], [34], [30] offers the image-based entities, such as light field [14], lumigraph [9], concentric mosaics [21] and panorama [5], an ability to change the illumination. The basic idea of IBR is to shift the intensive lighting computation to off-line. Hence, the rendering speed could be very fast.

IBR can be explained by the illumination adjustable image (IAI) approach [32], [30], [31], where the lighting information of a scene is described by a plenoptic illumination function (PIF). It is a collection of pre-recorded reference images under various light directions. Relighting the scene with complex illumination conditions is based on those pre-recorded reference images.

In IBR, there are two important issues. The first one is compression. As a raw IBR data set consists of many reference images, an efficient compression method is a must. Another issue is the rendering method. For complicated lighting conditions, the rendering speed is slow. For example, in the distant global illumination [11], we need to consider all incoming radiances. The brute-force way that integrates the contribution given by all incoming radiances is very time-consuming. Although we can apply the important sampling concept [29], [26] to simplify the integration process, the important sampling process on incoming radiances is still time-consuming.

For the compression issue, individually compressing each reference image with JPEG [28] is not efficient because there are strong correlations among reference images in the lighting domain. Using video compression methods [4] to process the IBR data set is also not efficient. This is because most current video compression methods process a set of images in a 3D manner, 2D in spatial domain and 1D in the time domain. There are several techniques [14], [16], [2] in compressing data for image-based navigation (the change of viewpoint) such as light field and concentric mosaics. For example, utilizing the disparity map, Magnor and Girod [16] proposed the MPEG-like codec for light field. However, the above methods [28], [4], [14], [16], [2] are not suitable to render a scene under complicated illumination conditions [12].

To address the above two issues, we can use the spherical harmonic (SH) approach [31], [13], [11] to represent the illumination property of a scene. The SH approach has been widely used in representing various spherical illumination properties, including bidirectional reflectance distribution function (BRDF) [20], [11], apparent BRDF (ABRDF) [31], [13], distant environment [3], and pre-computed radiance transfer (PRT) [23], [22], [24].

With the SH approach, a pixel in an IAI is represented by a SH coefficient vector. That is, an IAI is represented by a number of SH coefficient lightmaps which are floating-point images. Those lightmaps can be further compressed and then the final compressed data are stored in the secondary storage. For the global illumination, the complicated lighting environment is also represented by a SH vector. Hence, the outgoing radiance of a pixel is a dot-product of two SH vectors.

For a simple directional light source, the rendering process is very simple. The synthesized image is a weighted sum of SH lightmaps. The weighting factors are the SH basic function values. Since every pixel is associated with the same light direction, we only need to evaluate one set of SH basic function values. Hence, the rendering speed is very fast even though the SH basis functions are defined by computational intensive formulas.

However, for other kinds of complex light sources [30], such as point light source and slide projector effect, every pixel is associated with its own lighting direction. Hence, for every pixel, we need to evaluate its own set of SH basis function values. That means, the rendering speed could be very slow.

Efficient SH basis function evaluation can be achieved by using a table lookup technique. We can pre-compute SH basis function values and store them in graphics hardware as a number of cubemaps. Given a lighting direction, the SH basis function values can be instantly obtained from those cubemaps. However, modern graphics processing units (GPUs) have a limitation on the number of active textures and memory size. Those cubemaps consume memory, as well as, the scarce count of active textures.

To achieve an efficient and practical solution for high-frequency lighting effects, this paper discusses a GPU-friendly approach based on the spherical Gaussian kernel function (SGKF) [10], [13]. The homogeneity and simplicity of SGKF make the relighting process more efficient and practical for high-frequency lighting effects. In terms of approximation ability, our approach is comparable to the SH one but it consumes less hardware resource. Instead of discussing the compression [13], rendering methods for directional light source, point light source and slide projector are the main points of this paper.

The rest of this paper is organized as follow. The IAI representation is briefly discussed in Section 2. Section 3 describes the SGKF representation. Section 4 discusses practical GPU rendering implementations for directional light source, point light source and slide projector effect. Rendering results under the SH and SGKF approaches are presented in Section 5. Finally, a conclusion is drawn in Section 6.