Elsevier

Pattern Recognition

Volume 31, Issue 8, 1 August 1998, Pages 1033-1047
Pattern Recognition

Multi-image photometric stereo using surface approximation by Legendre polynomials

https://doi.org/10.1016/S0031-3203(97)00082-4Get rights and content

Abstract

This paper presents a photometric stereo algorithm that reconstructs object shapes from multiple images, in which given 3D surfaces are approximated by Legendre polynomials and the relationships between the given surface and its derivatives are represented in matrix forms in terms of a polynomial coefficient vector. The reflectance map is linearized and the cost function expressed in quadratic matrix form in terms of the polynomial coefficient vector is minimized. The relative depth and its derivatives are obtained by updating them iteratively. Computer simulation with various noiseless/noisy sets of test images shows that the performance of the presented two-image photometric stereo algorithm is comparable to that of the conventional methods in terms of three different error measures: brightness error, orientation error and height error. Also the performance comparison of the presented and conventional three-image photometric stereo algorithms for the noiseless/noisy sets of images is shown.

Introduction

Photometric stereo reconstructs 3D shapes from multiple images that are obtained at the same position with different light source positions. In general, the projection of a 3D real scene onto a 2D image plane results in loss of information such as depth or object shapes. The inverse procedure, i.e. reconstruction of 3D information from a 2D image restores lost information.[1]

The three-image photometric stereo problem[2] was expressed by the simultaneous first-order equations with three unknowns, assuming that source positions and the reflectivity function of an object were known. Coleman and Jain[3] tried to solve the problem with four images. They separated out specular and diffuse components from image brightness, and determined the surface normal for the diffuse component with three sources. Ikeuchi[4] used distributed light sources. Onn and Bruckstein[5] attempted to solve the problem with two images by using an integrability constraint. Kim and Burger[6] derived a single nonlinear equation by assuming the point source. Lee and Kuo[7] reconstructed the height with two images by employing an iterative modeling of 3D surfaces with finite triangular elements. Yang, Ohnishi and Sugie[8] reconstructed surface orientations with two images by using a convexity constraint.

In this paper, the photometric stereo problem is simplified by using several assumptions that are generally adopted in the shape from shading (SFS) problem.[2] For example, to simplify the problem, orthographic projection rather than perspective projection has been generally assumed. In addition, Lambertian reflectance is assumed, i.e. image brightness is assumed to depend only on p(x,y) and q(x,y), regardless of properties and reflectivity of an object, where p(x,y) and q(x,y) represent partial derivatives of the surface height z(x,y) with respect to x and y, respectively, i.e. p(x,y)=∂z(x,y)/∂x and q(x,y)=∂z(x,y)/∂y. The relationship between image brightness and surface orientations, with Lambertian assumption, can be expressed by the following nonlinear image irradiance equation:I(x,y)=R(p(x,y),q(x,y)),where I(x,y) represents image brightness at (x,y) and R(·) denotes a nonlinear function. The reflectance map R(p,q) is expressed asR(p,q)=η(N·S)=ηcosγ−pcosτsinγ−qsinτsinγ1+p2+q2where η denotes a constant albedo considering the strength of illumination and the reflectivity of the surface, independent of surface characteristics of an object and the properties of a light source employed. N=(−p,−q,1)/1+p2+q2 represents the surface normal vector at (x,y). S=(cosτsinγ,sinτsinγ,cosγ) means the unit vector for the source illumination, where τ denotes the angle between the projection of S onto the xy plane and the xz plane, and γ is the angle between S and the z-axis. It is assumed that the light source is located at infinity, thus its direction is the same at all points on the object surface.

In this paper, under the above assumptions, an approach to multiple-image photometric stereo is presented. First of all, in the presented multi-image photometric stereo algorithm, the surface height z is explicitly expressed by Legendre polynomials, then the relationship between z and its derivatives p and q are derived in matrix forms. Then common Legendre polynomial coefficients are computed from the relationships. The reflectance map is linearized, and is expressed in terms of polynomial coefficients. The polynomial coefficients obtained by minimizing the cost function expressed in quadratic matrix form is used to compute z,p and q, and in turn the surface height z is used to update the polynomial coefficients. An entire image is partitioned into a number of overlapping windows and the presented approach is applied to each window for reduction of the approximation error.

The rest of the paper is structured as follows. In Section 2, conventional photometric stereo algorithms are reviewed and their disadvantages are discussed. In Section 3, the concept of surface approximation by orthogonal polynomials is presented, then the multiple-image photometric stereo algorithm is introduced using surface approximation by Legendre polynomials. In Section 4, experimental results of the presented algorithm are shown for various sets of test images. We also show the performance comparison, for the noisy set of images, of the presented and conventional algorithms in terms of three different error measures. Finally, conclusions are given in Section 5.

Section snippets

Conventional photometric stereo algorithms

Most of photometric stereo and SFS problems use the reflectance map specified by the reflectance model with several assumptions. The reflectance map is expressed by the loci of contours having the same image brightness I in the gradient space. As shown in Fig. 1, the number of (p,q) pairs generating the same brightness is infinite. However, if we obtain two images generated by the sources at different positions, i.e. if we have two brightness values at a specific point of surfaces, two contours

Multi-image photometric stereo algorithm using surface approximation by Legendre polynomials

The height of object surfaces can be expressed as an explicit function z(x,y). In this section, a continuous approximation function z(x,y) is represented as a linear combination of orthogonal basis functions at discrete points of object surfaces. For example, the surface function z(x,y) defined over a 4×4 image is shown in Fig. 2. We regard this procedure as a linear transformation expressed in matrix forms. We compute coefficients of an approximation function, by which z,p and q are obtained.

Experimental results and discussions

We use 64×64 and 128×128 synthetic images to simulate the photometric stereo algorithm. We experiment with various image sets, each of which consisting of two or three input images. Sets of input images used in experiments are shown in Fig. 4, and its 3D shapes are shown in Fig. 5. As shown in Fig. 4, (tilt, slant) of a light source in generating synthetic images is set to (50°, 20°), (190°, 20°) and (250°, 20°). Fig. 4a shows a continuous sphere 1 image containing its peak at the center and Fig. 4

Conclusions

This paper presents a multi-image photometric stereo algorithm using surface approximation by Legendre polynomials. To reduce the blocking effect, an entire image is partitioned into a number of small overlapping windows. The relationship between z and p, q is expressed in matrix form in terms of the coefficient vector [Ĉ]. With the linearized reflectance map, the coefficient vector [Ĉ] is updated and p, q and z are iteratively updated. After obtaining the height function z(x,y), p and q are

Acknowledgements

This work was supported in part by ERC-ACI (of SNU) by KOSEF.

About the Author—BANG-HWAN KIM received the B.S. and M.S. degrees in electronic engineering from Sogang University, Seoul, Korea, in 1993 and 1995, respectively. Since 1995, he has been a research engineer at the DVD RD Center of Samsung Co., Ltd., Suwon, Korea. His research interests are computer vision and image coding.

References (13)

There are more references available in the full text version of this article.

Cited by (8)

  • Optimal illumination directions for faces and rough surfaces for single and multiple light imaging using class-specific prior knowledge

    2014, Computer Vision and Image Understanding
    Citation Excerpt :

    In [21] was shown that a plausible shape can be obtained based on two light sources and in [11] a multiple illumination technique that directly recovers the viewer-centered curvature matrix and being independent of knowledge of incident illumination orientation, local surface orientation, or diffuse surface albedo was presented. A scheme to resolve handwriting from background printing using photometric stereo to recover the surface was suggested in [25] and in [19] a photometric stereo algorithm was presented that reconstructs object shapes from multiple images, in which 3D surfaces were approximated by Legendre polynomials. In [37] an algorithm for shape from shading was introduced based on the assumption that a single input image will be matched to a second image through a uniform disparity field.

  • Image analysis and computer vision: 1998

    1999, Computer Vision and Image Understanding
  • Surface reconstruction by evaluating height from gradient values

    2014, Souvenir of the 2014 IEEE International Advance Computing Conference, IACC 2014
View all citing articles on Scopus

About the Author—BANG-HWAN KIM received the B.S. and M.S. degrees in electronic engineering from Sogang University, Seoul, Korea, in 1993 and 1995, respectively. Since 1995, he has been a research engineer at the DVD RD Center of Samsung Co., Ltd., Suwon, Korea. His research interests are computer vision and image coding.

About the Author—RAE-HONG PARK received the B.S. and M.S. degrees in electronic engineering from Seoul National University, Seoul, Korea, in 1976 and 1979, respectively. He received the M.S. and Ph.D. degrees in electrical engineering from Stanford University, Stanford, California, in 1981 and 1984, respectively. He joined the faculty of the Department of Electronic Engineering at Sogang University, Seoul, Korea, in 1984, where he is a currently professor. In 1990, he spent his sabbatical year at the Computer Vision Laboratory of the Center for Automation Research, University of Maryland, College Park, Maryland, as a visiting associate professor. His current research interests are computer vision, pattern recognition, and video communications. He is a member of the Pattern Recognition Society, IEEE and KITE.

View full text