3D facial shape reconstruction using macro- and micro-level features from high resolution facial images

Highlights

• Qualitative results of the 3D reconstructions for the proposed and the previous method are supplemented.

• Additional experiments for low-resolution images are supplemented.

• Dense 3D facial shape reconstruction procedures (Section 2.4) are revised according to the review's comments.

Abstract

Three-dimensional (3D) facial modeling and stereo matching-based methods are widely used for 3D facial reconstruction from 2D single-view and multiple-view images. However, these methods cannot realistically reconstruct 3D faces because they use insufficient numbers of macro-level Facial Feature Points (FFPs). This paper proposes an accurate and person-specific 3D facial reconstruction method that uses ample numbers of macro- and micro-level FFPs to enable coverage of all facial regions of high resolution facial images. Comparisons of 3D facial images reconstructed using the proposed method for ground-truth 3D facial images from the Bosphorus 3D database show that the method is superior to a conventional Active Appearance Model-Structure from Motion (AAM + SfM)-based method in terms of average 3D root mean square error between the reconstructed and ground-truth 3D faces. Further, the proposed method achieved outstanding accuracy in local facial regions such as the cheek—areas where extraction of FFPs is difficult for existing methods.

Introduction

Researchers have focused considerable attention on three-dimensional (3D) facial reconstruction technologies because they are useful in various applications, such as 3D person-specific games and movie-character generation [1], surgical 3D simulation, virtual glasses simulation, frontal facial synthesis of suspects in surveillance camera systems, 3D teleconferencing, and pose-invariant facial recognition [2], [3], [4]. Further, with the rapid development of image-capturing device technology, High Resolution (HR) images can now be obtained easily and economically. Consequently, 3D facial reconstruction research needs to encompass exploration of the micro-level information existing only in HR images in order to more accurately and realistically reconstruct 3D faces.

Conventional 3D facial reconstruction technologies can be classified into hardware-based and software-based approaches (as shown in Table 1).

Hardware-based approaches reconstruct 3D faces using additional devices such as multiple stereo cameras [5], structured light [6], 3D laser scanners [7], and depth sensors [8] (e.g., Microsoft Kinect). Although these devices can be used to reconstruct 3D faces accurately and realistically, they incur additional high costs. They also have portability problems because they are bulky and require some amount of space.

Software-based approaches reconstruct 3D faces using image sequences only. These approaches can generally be categorized into 3D Morphable Model (3DMM)-based methods [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], Structure from Motion (SfM)-based methods [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], and Shape from Shading (SFS)-based methods [34], [35], [36], [37], [38], [39], [40]. However, these methods cannot realistically reconstruct 3D faces because they use an insufficient number (approximately 80) of corresponding macro-level Facial Feature Points (FFPs). Further, they cannot extract good features for matching from facial sub-regions (e.g., cheekbone, forehead, moustache, and chin) from low-resolution images. Thus, the reconstructed 3D shapes in these regions are either biased to the mean facial shape or are not sufficiently accurate.

Much research has been conducted using micro-level features to recognize faces. Park et al. [41] and Klare et al. [42] analyzed three levels of features for face recognition. They suggested that Level 3 features—which they define as micro-level features such as moles, scars, freckles, pores, and birthmarks—are useful for distinguishing faces in HR images and can be extracted using Difference of Gaussian (DoG) or Laplacian of Gaussian (LoG) [43]. However, they could extract only a very limited quantity of micro-level features from human faces using their method. Thus, although it is possible to use these features for face recognition, it is difficult to accurately reconstruct 3D faces using only these features.

Li et al. [44], [45] proposed a novel pore-scale facial feature framework for verification of faces in HR images. By adapting the Scale Invariant Feature Transform (SIFT) detector and descriptor to their proposed pore-scale facial feature framework, and using a candidate-constrained matching scheme, the framework is able to establish a large number of reliable correspondences of keypoints between two face images of the same subject with dense and distinguishable facial features, which are desirable aspects for face verification. However, because their method does not match the corresponding points between two face images in each local facial region, outliers can be generated in other local regions, resulting in the computational costs for matching descriptors increasing.

To address the issues outlined above, we propose an accurate and user-specific 3D facial reconstruction method that utilizes ample numbers of FFPs in the various local facial regions from HR facial images. The primary contributions of this paper and the novel aspects of the proposed method are as follows (Table 1):

• The proposed method is the first to reconstruct 3D faces by combining macro-level features (e.g., eyes, eyebrows, nose, mouth, and facial contour) with micro-level features (e.g., pores, hairs, fine wrinkles, freckle, fine wrinkles, acne, and spots) from HR images. To the best of our knowledge, to date no other method that combines macro- and micro-level features to reconstruct 3D faces has been proposed in the literature.

• The performance of the proposed method, which utilizes ample numbers of corresponding FFPs to reconstruct 3D faces, is better than that of previous sparse correspondence-based 3D reconstruction methods, which use an insufficient number of corresponding FFPs from macro-level features. Further, the proposed method achieves excellent accuracy in the local facial regions, such as cheeks, forehead, philtrum, and chin, from which FFPs are difficult to extract.

The remainder of this paper is organized as follows. Section 2 presents the proposed 3D facial reconstruction method. Section 3 outlines the experiments conducted and analyzes the results obtained. Section 4 presents our conclusions along with plans for future work.