Facial modeling from an uncalibrated face image using a coarse-to-fine genetic algorithm

Abstract

This paper presents a genetic algorithm-based optimization approach for facial modeling from an uncalibrated face image using a flexible generic parameterized facial model (FGPFM). The FGPFM can be easily modified using the facial features as parameters of FGPFM to construct an accurate specific 3D facial model from only a photograph of an individual with a yawed face based on the projection transformation. The facial modeling problem is formulated as a parameter optimization problem and the objective function is also given. Moreover, a coarse-to-fine approach based on our intelligent genetic algorithm which can efficiently solve the large parameter optimization problems is used to accelerate the search for an optimal solution. Furthermore, sensitivity analysis and experimental results with texture mapping demonstrate the effectiveness of the proposed method.

Introduction

Face images have received considerable attention, particularly in the fields of computer vision and signal processing communities. For instance, model-based image coding methods have been proposed for future videophone and video conference services. However, the images in these applications are complex and highly variable, even for a specific individual. An important problem is how to create a 3D model of a specific individual. Automatic creation of a 3D facial model of a specific individual plays an important role in many applications, such as model-based coding for narrow-band visual communication [1], [2], [3], [4], view-independent face recognition tasks [5], [6], and image synthesis problems in areas like virtualized reality [7] and synthesis of novel views [8], [9].

3D facial models can be categorized into two classes: those based on the view-independent 3D facial structure and those considering only view-dependent facial models. The view-dependent facial model uses multiview representation in which a set of 2D image-based example facial models are combined into a flexible 3D facial model by a weighted sum of given example facial models [9], [10]. The limitations of view-dependence narrow the scope of 3D facial model-related applications. The approaches used to automatically create 3D facial models with a view-independent 3D facial structure can be applied more extensively [3]. Approaches capable of creating a view-independent 3D facial model of a specific individual can be categorized into two groups: use of an actual 3D face of a specific individual and use of a generic facial model with 2D face images of an individual. The approaches that need an actual 3D face include active vision [2], 3D digitizer [8], and vision-based methods [11].

Approaches belonging to the second group in which the 3D facial model of a specific individual is constructed consist of two steps. Firstly, select a generic model representing the topological structure of a typical face and typical 2D face images of the individual. Secondly, adjust the geometrical shape of the generic model to that of the actual face images through 3D transformation and modification according to the positions of some facial features such as eyes, mouth, nose, and facial contour. Akimoto et al. [1], Tang and Huang [12], and Ip and Yin [7] used two orthogonal face images of an individual's head to acquire the features deemed necessary for fitting a generic model to an individual's head. Pei et al. [4] used the transformation of the generic facial model with an affine mapping for model-based image coding by tracking 3D contour feature points. Aizawa et al. [14] used multiple face images to adjust a flexible three-dimensional facial model for a particular face. Eisert and Girod [15] changed the texture and control points’ position using information from 3D laser scans to adjust the generic 3D model to a specific individual.

Given only a photograph of an individual's face shown in front-viewed or as a randomly yawed face when viewing the face from close range, can one automatically create an accurate 3D facial model of the individual? For this purpose, assessing the effectiveness of the above approaches is relatively difficult since no work has reported on the generalization performance to automatically create 3D facial models that takes the pose of the face and the death information in the fitting process into consideration from only a photograph.

Luo and King [13] developed a facial-feature-extraction algorithm to automate the process of fitting the general facial wire frame model to the actual face image. Furthermore, by introducing face orientation detection during the fitting process, the facial model construction method is robust with respect to its ability to adjust the specific facial 3D model for arbitrary orientations. However, the depth values of the facial model were adjusted in the z-direction by an amount proportional to its change in 2D image plane [13]. A weak perspective imaging process is assumed valid when depth changes on the face are small compared with the long distance between the face and the camera. This is generally a good approximation, except when viewing the face from close range with a short focal length lens, which results in significant perspective distortion in the image [35]. To adjust the 3D control points of a generic facial model to fit the face image accurately, the pose of the yawed face must be determined and depth information of facial features must be taken into account simultaneously, especially when viewing the face from close image. How to accurately modify the generic facial model to fit the specific face image from an uncalibrated face image using the perspective imaging process is investigated in the work.

Two fundamental problems which must fully cooperate with each other are the model modification method and the establishment of the generic facial model, described as follows.

In the light of the above two problems, this paper presents a GA-based optimization approach for facial modeling from an uncalibrated face image using a flexible generic parameterized facial model (FGPFM). The microstructural information can be expressed using the structural FGPFM with representative facial features that can be accurately found in the image. The reconstruction procedure can be regarded as a block function of FGPFM, and the input parameters are the 3D face-centered coordinates of control points. Once the control points are given, the desired 3D facial model is determined based on the topological and geometric descriptions of FGPFM. How to reconstruct the 3D facial model is transformed into a problem of how to acquire the accurate 3D control points.

Since the solution space is large and complex considering the large number of control points in 3D space, the proposed coarse-to-fine approach based on our intelligent genetic algorithm IGA [31] is used to efficiently solve the optimization problem. IGA is an efficient general-purposed algorithm capable of solving large parameter optimization problem. Coarse-to-fine approach can efficiently adjust control points in 3D space. The fitness function takes into account the evidence from the face image and human perception. The proposed coarse-to-fine IGA can effectively construct an optimal facial model. Merits of the proposed method are summarized as follow. (1) FGPFM is presented so that the good parameters, the control points, of FGPFM can yield the good facial model for a specific person. (2) An analytic solution for the pose determination of human faces (PDF) [32] from a monocular image is applied to obtain the initial 3D control points and make coarse-to-fine IGA more efficient. (3) The reconstruction problem is formulated as a parameter optimization problem based on the ability of FGPFM and PDF. Furthermore, the coarse-to-fine IGA is also used to speed up the search for an optimal solution which is a set of optimal control points. The facial model construction method can obtain more accurate facial models with respect to its uses of the perspective fitting process and the coarse-to-fine IGA for adjusting the 3D control points of FGPFM, compared with that of the existing ones, e.g. Ref. [13].

The rest of this paper is organized as follows. Section 2 describes how to establish a flexible generic parameterized facial model. Section 3 formulates the facial modeling problem as an optimization problem and also outlines the reconstruction procedure. Section 4 summaries the sensitivity analysis and shows the experimental results with texture mapping. Conclusions are finally made in Section 5.