3D face reconstruction from skull by regression modeling in shape parameter spaces
Introduction
A 3D face reconstruction is fundamentally important for practical applications in face recognition and retrieval [1], [2], [3]. Research on anthropology [4] indicates that human face shape is mainly determined by the skull. A 3D face reconstruction from skull, also called craniofacial reconstruction [5], is to estimate a person׳s face from its skull using the relationship between facial soft tissues and the underlying bone structure. Unlike 3D face reconstruction in the usual sense [2], [6], it reconstructs the 3D face model not from 2D face images but from the skull. Craniofacial reconstruction can be applied in many fields such as forensic medicine [7], archaeology [8], face animation [9]. For example, it can provide a clue and trigger recognition for an unknown victim׳s skull with no other evidence available. Until now, many techniques have been investigated for craniofacial reconstruction, but it is still a hard task as accurate extraction and representation of the intrinsic relationship between faces and skulls has not been well solved.
Traditionally, craniofacial reconstruction [4], [7] was manually conducted by anatomists or artists. They model a face by adding clay or plasticine to a skull replica relying on their experience. The process is time consuming and subjective. Different practitioners usually produce extremely different results. With the rapid development of 3D digitization and computer graphics technologies, computer-aided craniofacial reconstruction methods that deal with digitized models come forth. They can overcome the drawbacks of manual methods. Most computerized methods [5] share a similar idea that the face of a target skull can be obtained by deforming a generic or a specific craniofacial template chosen by skull similarities or properties. Some techniques [8], [10] fit a head template to a set of interactively placed virtual dowels at the target skull, which are obtained by adding statistical tissue thickness values to corresponding skull landmarks. Others [11], [12], [13] deform a reference skull to the target skull, and then apply an extrapolation of the obtained skull deformation to the reference facial surface. As these methods ignore personal variation of the tissue thickness and assume that all human faces of the same ethnic group or at least the reference and the target have similar tissue thickness distributions, they cannot well explain craniofacial morphology variation. Consequently, the template-related model bias exists. In addition, the applied generic deformations always result in unrealistic, caricature-like, or implausible facial reconstructions if difference between the template or reference skull and target skull is large [14].
Nowadays, machine learning methods have been widely used in 3D face reconstruction and retrieval [2], [15]. Inspired by this, we propose to learn the relationship between skulls and faces by regression modeling. Instead of deforming a single reference template, we build statistical shape models for skulls and faces, and establish regression models in the shape parameter spaces. To better represent the craniofacial shape variation and boost the reconstruction, we adopt a regional modeling and reconstruction strategy. Moreover, the attributes such as age and BMI can be easily added to obtain the face reconstruction with different attributes.
The article is organized as follows: Related works are introduced in Section 2. The statistical shape models are presented in Section 3. Craniofacial reconstruction is described in Section 4. Experimental results are shown in Section 5. Conclusions are given in the end.
Section snippets
Related works
Similar to our work, many techniques try to learn the statistical shape relationship between the face and skull from multiple reference heads, and the facial surface is reconstructed by using the statistical relationship. Most of them build a statistical deformable model by principal component analysis (PCA). Berar et al. [16], [17] established a joint statistical deformable model of the face skin and skull, both of which are represented as sparse meshes with thousands of vertices. The face
Database
We have obtained a database of 114 whole head CT scans on voluntary persons who mostly come from Han ethnic group in the North of China, aged 20–60 years: 52 females and 62 males. The CT images were obtained using a clinical multi-slice CT scanner system (Siemens Sensation 16) in the Xianyang hospital located in western China. Each head of our database consists of a skull surface and its corresponding face skin surface, which are extracted from the CT images. All the heads are substantially
Craniofacial reconstruction based on PLSR
PLSR is a powerful multivariate data analysis technique [29] for problems where the data are noisy and highly correlated, and where there are a limited number of observations. It projects input and output variables into a subspace of orthogonal latent variables, respectively, and relates them by means of regression models between the latent variables. Each pair of latent variables occupies a certain amount of variability of the input and output data sets. We use PLSR to explore the shape
Experimental results
The database described in Section 3 is divided into a training set and a test set. The training set includes 96 pairs of skull and skin surfaces in front, and the test set contains other 18 pairs. The holistic regression model and the regional regression models are established as Section 4 describes, where the mode number of the statistical shape models (i.e., in Eqs. (1), (2)) is determined by the variance contribution rate of 98%. Throughout the experiment, the mode numbers are 39 and 19
Conclusions
A 3D face reconstruction from skull has been practiced for over one hundred years. It can be applied in many fields such as forensic, archaeology. Traditional manual reconstructions need a high degree of anatomical or artistic expertise, and the reconstruction results are subjective and variable. Current computerized methods mainly concentrate on deforming a craniofacial template, which always leads to template-related model bias. Most statistical learning based methods use sparse landmarks or
Acknowledgments
We thank the anonymous reviewers for their helpful comments. This work was partially supported by the National Natural Science Foundation of China (Grant No. 61272363) and Program for New Century Excellent Talents in University (NCET-13-0051).
Fuqing Duan received his Ph.D. degree in 2006 from the National Laboratory of Pattern Recognition (NLPR) at the Institute of Automation of the Chinese Academy of Sciences (CAS), China. Currently, he is an associate professor at the College of Information Science and Technology, Beijing Normal University. His research interests are computer vision and pattern recognition.
References (29)
- et al.
Computerized craniofacial reconstruction: conceptual framework and review
Forensic Sci. Int..
(2010) - et al.
Facial reconstruction using 3-D computer graphics
Forensic Sci. Int.
(2000) - et al.
Computerized craniofacial reconstruction using CT-derived implicit surface representations
Forensic Sci. Int.
(2006) - et al.
Bayesian estimation of optimal craniofacial reconstructions
Forensic Sci. Int.
(2010) - et al.
Craniofacial reconstruction as a prediction problem using a latent root regression model
Forensic Sci. Int.
(2011) - et al.
A local technique based on vectorized surfaces for craniofacial reconstruction
Forensic Sci. Int.
(2010) - et al.
Non-linear PLS modelling
Chemometr. Int. Lab.
(1989) - et al.
Face matching and retrieval in forensics applications
IEEE Multimedia
(2012) - et al.
Three-dimensional face reconstruction from a single image by a coupled RBF network
IEEE T. Image Process.
(2012) - et al.
3-D Object retrieval and recognition with hypergraph analysis
IEEE T. Image Process.
(2012)
The Face Finder
Forensic Facial Reconstruction
Reanimating the dead: reconstruction of expressive faces from skull data
ACM. T. Graph. (TOG)
Cited by (43)
Bidirectional prediction of facial and bony shapes for orthognathic surgical planning
2023, Medical Image AnalysisCraniofacial reconstruction based on heat flow geodesic grid regression (HF-GGR) model
2021, Computers and Graphics (Pergamon)3D facial expression modeling based on facial landmarks in single image
2019, NeurocomputingHow tight is the relationship between the skeletal and soft-tissue facial profile: A geometric morphometric analysis of the facial outline
2018, Forensic Science InternationalCitation Excerpt :In manual facial approximation methods these relationships are described as prediction guidelines, they are based on statistical analysis of linear or angular measurements or observations and focused especially on estimation of size and shape of facial organs (eyes, nose, mouth and ears), e.g. [8–13]. In computerized methods, the anatomical relation of face to skull (CFI — craniofacial information) is contained in craniofacial model (CFM) and rises from knowledge of facial surfaces, skull surface, facial muscles, soft tissues thickness learned from a large and diverse facial reference database [4,14–17]. According to Stephan and Henneberg [18], facial approximations infrequently result in specific and purposeful facial recognition.
Fuqing Duan received his Ph.D. degree in 2006 from the National Laboratory of Pattern Recognition (NLPR) at the Institute of Automation of the Chinese Academy of Sciences (CAS), China. Currently, he is an associate professor at the College of Information Science and Technology, Beijing Normal University. His research interests are computer vision and pattern recognition.
Donghua Huang received her M.S. degree in 2012 from College of Information Science and Technology, Beijing Normal University. Currently, she is a lecturer at the College of Navigation and Aerospace Engineering, The PLA Information Engineering University.
Yun Tian received his Ph.D. degree in Signal and Information Processing from Northwestern Polytechnic University, China, in 2007. Currently, he is an associate professor at the College of Information Science and Technology, Beijing Normal University. His research interests include pattern recognition and medical image processing.
Ke Lu received his M.S. and Ph.D. degrees from the Department of Mathematics and the Department of Computer science at Northwest University in July 1998 and July 2003, respectively. He then worked as a postdoctoral fellow in Institute of Automation Chinese Academy of Sciences from July 2003 to April 2005. Currently, he is a professor of University of the Chinese Academy of Sciences. His research areas mainly focus on curve matching, 3D image reconstruction, and computer graphics.
Zhongke Wu is full professor and Ph.D. student supervisor in College of information science and technology, Beijing Normal University, China. Prior to joining in BNU, he worked in Nanyang Technological University (NTU), Singapore, Institute National de Recherche en Informatique et en Automatique (INRIA) in France, Institute of High Performance Computing (IHPC), and Institute of Software, Chinese Academy of Sciences, in China, from 1995 to 2006. His research interests are image processing and computer graphic.
Mingquan Zhou is a professor and doctoral supervisor at the College of Information Science and Technology, Beijing Normal University, and also the dean of this college. His research interests are computer graphic and virtual reality.