Uniformly sampling multi-resolution analysis for image-based relighting

https://doi.org/10.1016/j.jvcir.2010.05.004Get rights and content

Abstract

Image-based relighting allows us to efficiently light a scene under complicated illumination conditions. However, the traditional cubemap based multi-resolution analysis unevenly samples the spherical surface, with a higher sampling rate near the face corners and a lower one near the face centers. The non-uniformity penalizes the efficiency of data representation. This paper presents a uniformly sampling multi-resolution analysis approach, namely the icosahedron spherical wavelets (ISW), for image-based relighting under time-varying distant environment. Since the proposed ISW approach provides a highly uniform sampling distribution over the spherical domain, we thus can efficiently handle high frequency variations locally in the illumination changes as well as reduce the number of wavelet coefficients needed in the renderings. Furthermore, visual artifacts are demonstrated to be better suppressed in the proposed ISW approach. Compared with the traditional cubemap based multi-resolution analysis approach, we show that our approach can effectively produce higher quality image sequences that are closer to the ground truth in terms of percentage square errors.

Introduction

Image-based relighting (IBL) [1], [2], [3], [4] offers image-based entities [5], [6], [7], [8] an ability to change the illumination. Since no geometry is used during the rendering, the rendering speed is scene-independent. Hence, IBL was quickly adopted in a wide range of high performance applications [9], [10].

Wong [2], [4] proposed the plenoptic-illumination function (PIF) to represent an illumination adjustable image (IAI), which allows us to control the lighting conditions of a scene. The raw data of an IAI are a number of reference images captured under variable lighting directions. The collection of reference images is an IBL data set, which can, in turn, be interpreted as a 4D function: 2D in the spatial domain (rectangular system) and 2D in the lighting domain (spherical surface).

In IBL, there are two issues. The first one is the rendering method. For complicated lighting conditions, such as the illumination by distant environments [11], we need to consider all incoming radiances that are usually represented by a discrete spherical function; using a brute-force way to integrate all incoming radiances is, however, extremely time-consuming. Hence, an efficient rendering solution for handling complex lighting conditions is thus highly desirable. The second one is data compression. As a raw IBL data set consists of many reference images, an efficient compression methods is a must. However, using conventional image/video compression methods [12], [13] or multi–dimensional compression methods [14], [15] is also insufficient because those compression methods do not fully consider the correlation of the data in the lighting domain. Other than that, there are several techniques [5], [16], [17] in compressing data for image-based navigation, such as light field and concentric mosaics. These methods, however, are not designed for handling IBL data sets, so they cannot efficiently handle incoming radiances from all directions, as in environment-based illumination [11].

To address the above two issues (data compression and rendering), we can use the spherical harmonic (SH) approach [11], [18], [19], [20], [21] to represent the illumination variation. But similar to the Fourier series or transform, the SH approach is very weak in representing local high frequency lighting features, such as shadow and highlight.

To enable high frequency lighting features, Ng et al. [22] proposed a cubemap-based spherical wavelet (CSW) approach. A pixel in an IAI is then represented as a number of CSW coefficients, while the same CSW transform is also applied to the distant environment. Afterwards, only important wavelet coefficients are used for rendering. Nevertheless, using cubemap to represent spherical functions creates an uneven sampling problem. That is, face centers of the cubemap have much lower sampling density than face edges and corners. Thus, some lighting features near face centers could be represented in a very poor manner. Fig. 1 shows rendered images of a teapot illuminated by a distant environment that contains an area light source from above (above the center of the upper cubemap face). In Fig. 1(a), we use a very high resolution uniformly-distributed sampling pattern so as to obtain a highly realistic controlled image. When we use a 6 × 32 × 32 cubemap (6144 samples) to represent the spherical functions, the rendered image (as shown in Fig. 1(b)) from the CSW contains undersampling artifact in the shadow. As might be expected, if we increase the resolution of the cubemap, the artifacts can be suppressed, but this trivial solution will also increase the computational and space complexities in both the rendering and data preparation. Besides, our experiments also find out that the number of CSW coefficients could be very large.

Note further in the CSW approach that regions near the face corners have relatively higher sampling density, thus the quality of rendered images in these regions could be very good. However, more wavelet coefficients are involved because the sampling is denser in these regions. In other words, face centers, edges, and corners all have different (unbalanced) rendering quality as well as different number of wavelet coefficients in use. Such an inhomogeneity makes the rendering complex and inefficient. Although one may suggest that if we can rotate the environment map and then shift the spot light to the corner of the cube, the CSW approach may perform better. However, this approach can only work for some limited situations, such as static environment maps or environment maps with a single spot light. For a time-varying distant environment with multiple high frequency light sources, we actually cannot control light sources’ positions. Hence, for time-varying distant environment, using a uniformly sampling spherical wavelet for rendering is of paramount important.

There are some works [23] of using icosahedron spherical wavelet (ISW) [24], [25], [26] in vision and graphics communities. Paster and Rodriguez [23] used the icosahedron based sampling points to represent the surface of an 3D object. In their representation, coarser shape can be represented by the scaling coefficients obtained after a spherical wavelet transform. With the spherical wavelet transform, the 3D object can be represented in a multi-resolution manner. In [26], a more practical spherical Haar wavelet basis that is both orthogonal and symmetric was proposed. There are also some previous works [27], [28] related to icosahedron for rendering images. However, none of these works demonstrated the advantages of using the uniformly sampling property of an icosahedron. Also, there is no performances comparison between the ISW and CSW approaches.

This paper discusses how to use the ISW approach for illuminating image-based entities under time-varying distant environment. In our approach, we use ISW to represent an IAI and use the dual ISW to represent the time varying distant environment. Afterwards, only important wavelet coefficients are used for rendering. Since the icosahedron sample points are more uniformly distributed on the sphere, we can efficiently and effectively handle local high frequency variations over the 4Π spherical domain, and also reduce the number of wavelet coefficients required.

Compared with the CSW approach, our approach can produce higher quality image sequences that are closer to the controlled images generated exhaustively from the raw data. Furthermore, since our representation is local, compact, and efficient, we can effectively encode sharp illumination changes. Fig. 1(c) shows the rendering result of using the ISW. Note that there are only 5120 sample points on the icosahedron. From the figure, we can see that undersampling artifacts are suppressed.

The organization of this paper is as follows: Section 2 discusses the concept of illumination model. Section 3 introduces the mathematical formulation of the ISW and its dual. In addition, the modeling of ISW from IAI and dual ISW environment coefficient vector sequence are also presented. Furthermore, Section 4 presents the efficient rendering method driven by important dual ISW environment coefficients, while Section 5 shows the simulation results. Finally, Section 6 draws the conclusion.

Section snippets

The illumination adjustable image (IAI)

The concept of the IAI is based on the plenoptic-illumination function (PIF) [29], [19], [30]. To simplify our discussion, we consider perspective views. A PIF,P(x,y,ω),tells us the radiance value of a pixel located at position (x, y) on the view-plane when the scene/object is illuminated by a directional light from ω, see Fig. 2.

The exact form of a PIF is usually unknown, but sampling the values in a PIF can be done fairly. In essence, sampling a PIF can be considered as taking pictures under

Scaling functions and wavelet functions

The ISW [23] is a spherical wavelet with a highly uniform sampling pattern. As shown in Fig. 7, each of the 20 base triangles on the icosahedron occupies an area of spherical triangle on the sphere, and each spherical triangle can be further subdivided into four smaller spherical triangles.

At the jth subdivision level, a spherical triangle is denoted as Tj,k, where k  K(j) = {0, 1,  , 20 × 4j  1} is the index to the kth spherical triangle at level j. Note that j = 0 means that there is no subdivision and

Important coefficient rendering

After performing the ISW analysis on both the IAI and the distant environment, Eq. (3) can be implemented readily in the ISW coefficient domain, given byLout(x,y,t)=i=1MP(x,y,ωi)L(ωi,t)=i=1Mzi(t)wi(x,y),where zi(t) and wi(x, y) denote the ith element in the corresponding coefficient vector.

In terms of implementation, both Eqs. (3), (16) can be applied to rendering, and their computational efficiency appears to be the same because both involves two M-dimensional vectors. However, unlike the

Simulation results

In the experiment, we compare our ISW approach with the CSW approach [22]. Two IAI data sets, Teapot and Bunny, are used as the testing data. Both are sampled using sampling distributions: cubemap points and icosahedron points. For cubemap points, the resolution of each face is 32 × 32, and hence, the number of sample points is 6144. For icosahedron points, the subdivision level equals to 4, and hence, the number of sample points is 5120. The sample images are sub-sampled from a very dense data

Conclusion

The conventional cubemap-based spherical wavelet (CSW) approach offers several advantages over previous image-based relighting methods such as the SH approach. Local high frequency features like shadow and highlight in distant environment illumination can be properly included in the renderings, and it can also provide an efficient rendering scheme that allows us to employ very few important wavelet coefficients during the rendering. The CSW approach is an advanced rendering technique widely

Acknowledgments

This work was supported by General Research Funds from Hong Kong Government (Project No.: CityU 116508) and (Project No. CUHK417107).

Ping-Man Lam received the B.Eng. and Ph.D. degree in electronics from the City University of Hong Kong in 2002 and 2007, respectively. His research interest is computer graphics.

References (32)

  • Z. Wang et al.

    Data compression with spherical wavelets and wavelets for the image-based relighting

    Computer Vision and Image Understanding

    (2004)
  • P. Debevec

    Image-based lighting

    IEEE Computer Graphics and Applications

    (2002)
  • T.-T. Wong, P.-A. Heng, S.-H. Or, W.-Y. Ng, Image-based rendering with controllable illumination, in: Proceedings of...
  • Y. Yu, J. Malik, Recovering photometric properties of architectural scenes from photographs, in: SIGGRAPH ’98:...
  • T.-T. Wong et al.

    Interactive relighting of panoramas

    IEEE Computer Graphics and Applications

    (2001)
  • W.-C. Chen, J.-Y. Bouguet, M.H. Chu, R. Grzeszczuk, Light field mapping: efficient representation and hardware...
  • S.J. Gortler, R.Grzeszczuk, R.Szeliski, M.F. Cohen, The lumigraph, in: SIGGRAPH 1996: Proceedings of the 23rd Annual...
  • H.-Y. Shum, L.-W. He, Rendering with concentric mosaics, in: SIGGRAPH 1999: Proceedings of the 26th Annual Conference...
  • S.E. Chen, Quicktime VR: an image-based approach to virtual environment navigation, in: SIGGRAPH 1995: Proceedings of...
  • B. Choudhury, S. Chandran, A survey of imagebased relighting techniques, in: GRAPP2006, 2000, pp....
  • M. Pharr et al.

    Physically-Based Rendering: From Theory to Implementation

    (2004)
  • J. Kautz, P.-P. Sloan, J. Snyder, Fast, arbitrary BRDF shading for low-frequency lighting using spherical harmonics,...
  • G.K. Wallace

    The JPEG still picture compression standard

    Communications of the ACM

    (1991)
  • B. Usevitch

    A tutorial on modern lossy wavelet image compression: foundations of JPEG 2000

    IEEE Signal Processing Magazine

    (2001)
  • P.L. Dragotti et al.

    Compression of multispectral images by three-dimensional SPIHT algorithm

    IEEE Transactions on Geoscience and Remote Sensing

    (2000)
  • B.-J. Kim et al.

    Low bit-rate scalable video coding with 3-D set partitioning in hierarchical trees (3-D SPIHT)

    IEEE Transactions on Circuits and Systems for Video Technology

    (2000)
  • Cited by (1)

    Ping-Man Lam received the B.Eng. and Ph.D. degree in electronics from the City University of Hong Kong in 2002 and 2007, respectively. His research interest is computer graphics.

    Chi-Sing Leung received the B.Sci. degree in electronics, the M.Phil. degree in information engineering, and the Ph.D. degree in computer science from the Chinese University of Hong Kong in 1989, 1991, and 1995, respectively. He is currently an Associate Professor in the Department of Electronic Engineering, City University of Hong Kong. His research interests include neural computing, data mining, and computer graphics. In 2005, he received the 2005 IEEE Transactions on Multimedia Prize Paper Award for his paper titled, “The Plenoptic Illumination Function” published in 2002. He was a member of Organizing Committee of ICONIP2006. He is the Program Chair of ICONIP2009. He is also a governing board member of the Asian Pacific Neural Network Assembly (APNNA).

    Tien-Tsin Wong received the B.Sci., M.Phil., and Ph.D. degrees in computer science from the Chinese University of Hong Kong in 1992, 1994, and 1998, respectively. Currently, he is an Associate Professor in the Department of Computer Science and Engineering, Chinese University of Hong Kong. His main research interest is computer graphics, including image-based rendering, natural phenomena modeling, and multimedia data compression. He received the IEEE Transactions on Multimedia Prize Paper Award 2005 and the Young Researcher Award 2004.

    Chi-Wing Fu received the B.Sci. and M.Phil. degrees in computer science and engineering from the Chinese University of Hong Kong, Shatin, Hong Kong, in 1997 and 1999, respectively, and the Ph.D. degree in computer science from Indiana University Bloomington in 2003. He is currently an Assistant Professor in the School of Computer Engineering at Nanyang Technological University, Singapore. His research interests include tile-based modeling and rendering methods, image-based rendering, texture synthesis, and large-scale astrophysical visualization. Dr. Fu received the IEEE Transactions on Multimedia Prize Paper Award in 2005.

    View full text