Joint gender, ethnicity and age estimation from 3D faces: An experimental illustration of their correlations☆
Introduction
In daily life, human beings perform gender and ethnicity recognition as well as estimate the age of their peers naturally and effectively. Several studies from different backgrounds (face and head anthropometry, cognitive psychology, clinical studies, etc.) have tried to understand how the process works. In particular, a number of anthropometric studies [65] have revealed that significant facial morphology differences exist among the gender, the ethnicity and the age groups. For example, when studying the Sexual Dimorphism (Male/Female differences) [9], researchers have found that male faces usually possess more prominent features than female faces. Male faces usually have more protuberant noses, eyebrows, more prominent chins and jaws. The forehead is more backward sloping, and the distance between top-lip and nose-base is longer. [65] have also demonstrated that all the concerned anthropometric measurements of females are smaller. In the study of the ethnic differences [16], researchers have found that compared to the North America Whites, Asians usually have broader faces and noses, far apart eyes, and exhibit the greatest difference in the anatomical orbital regions (around the eyes and the eyebrows). In the clinical study reported in [35], Alphonse et al. have revealed that Caucasians have significantly lower fetal Fronto-Maxillary Facial Angle (FMFA) measurements than Asians. In [65], sixteen anthropometric measurements have been recognized as significantly different between Asian and Caucasian faces. When studying the face aging [48], [49], researchers have concluded that the craniofacial growth is the main change in baby and adolescent faces, which results in the re-sizing and redistribution of facial features. During this period, generally, the larger the age, the bigger is the size of the face. When the craniofacial growth stops at 18–20 years old, the face contour and texture changes become the dominant changes. Young adults tend to have a triangle shaped face with small amount of wrinkles. In contrast, old adults are usually associated with a U-shaped face with significant wrinkles on the face. Besides the existence of these Soft-Biometric Traits2 [11], [34] in the face, gender, ethnicity and age are also correlated in characterizing the facial shape [65]. For example, according to the anthropometric studies cited above, the shape of the nose is influenced by all the three soft-biometric traits. In human perception, female faces usually look smoother and younger than male faces, and the Asian faces usually look younger than Non-Asian faces [50]. In [58], Vignali et al. have demonstrated both visually and quantitatively that ethnicity and gender are correlated to some extend in the 3D face. In [19], Gao et al. have concluded that the gender classifier trained on a specific ethnicity could not get good generalization ability on other ethnic groups. In [58], when the gender information is removed from the faces, the human ethnicity classification performance is recognizably lower.
In this paper, we consider the problem of joint gender, ethnicity and age recognition through the morphological differences in the 3D facial shapes. A number of morphology-driven 3D features are extracted, and then used to train Random Forest in Classification/Regression modes for gender and ethnicity recognition, and age estimation. A quantitative study of the correlations of the three demographic traits is proposed. In the joint recognition experiments, these correlations are considered, to demonstrate their influences on the recognition accuracy.
Since the 90s of the previous century, several approaches have been proposed to solve the problem of image-based automatic facial Soft-Biometric Traits recognition, e.g. gender [22], [26], [45] , ethnicity [17], [26] and age [18], [22], [26], [48]. While the most conventional works have attempted to exploit the intensity (color) images, a recent trend consists of investigating the use of the 3D shape of the face. According to the definition given in [62], the face texture represents the reflection and absorption effects of external illumination caused by the facial skin, while the 3D face shape defines the solid border which distinguishes the face and the environment. It is now well-established that the 3D shape provides a rich description of the face morphology compared to the intensity image. From 2D intensity images, the (2D) shape of the face is usually represented by a sparse set of anchor points detected in the face images (used to define the well-known Active Shape and Active Appearance Models), which represent poorly and incompletely the facial shape, and are sensitive to the head pose changes during the image acquisition. In the study presented in [29], Hu et al. have demonstrated that, with the 3D shapes of the face, human observers perform better on both gender and ethnicity recognition than with the 2D images.
With a particular focus on the 3D shape-based methods, this paragraph provides a brief review of the existing work in gender, ethnicity and age recognition. We adopt the taxonomy proposed recently in [26] which consists of three categories: (1) anthropometry-based, (2) Image-based, and (3) Appearance-based in automatic demographic traits recognition. The recent work of Gilani et al. [21] belongs to the first category (anthropometry) as it has proposed to automatically detect the biologically significant 3D facial landmarks, and then calculate the Euclidean and the Geodesic (along the surface) distances between them as face features. Similar studies have been proposed in [58], [27] where the 3D facial landmarks coordinates [58], and the volume/area information of facial regions [27] are extracted for gender and ethnicity classification. Despite their high performance, these approaches require accurate detection of the anchor points on the face. The works of Toderici et al. [54], and Wang and Kambhamettu [59] belong to the second category where image-based features like the wavelets [54], the Local Binary Patterns (LBP) [59], and the Shape Index [59] are extracted from the range images for gender and ethnicity recognition. With 3D facial meshes, Tokola et al. [55] have extracted the Correspondence Vectors (CVs) features to recognize all the three facial traits. Approaches which combine 2D (texture) and 3D (shape) channels form the third category (appearance-based). In [30], Huang et al. have combined the Local Circular Patterns (LCP) features of texture and range images in gender and ethnicity recognition. Also in [60], Wu et al. have fused shape and texture information implicitly with needle maps recovered from intensity images. In [33], Huynh et al. have fused the Gradient-LBP from range images and the Uniform LBP features of the gray image in gender classification. In [44], Lu et al. have fused the SVM outputs from range and intensity images in gender and ethnicity classification. Reported results of the appearance-based demonstrate higher accuracy than using only the intensity or the depth. However, when using the 2D intensity or range images, manually labeled facial landmarks are usually required in a pre-processing step to crop and align the facial regions [26], [30], [44].
From the analysis above, most of the existing 3D-based methods have tried to extract conventional features from range images (such as the 3D landmark coordinates, LBP, LCP, wavelets, shape index). No attention has been paid to reveal the relationship between the extracted features and the studied demographic trait. That's to say, even an approach achieves high recognition performance, we don’t know why the extracted features are relevant in the studied task. Moreover, the study of these soft-biometric traits has been done separately, in the 3D domain. The correlations of the soft-biometric traits have attracted little consideration. Although some 2D-based works have investigated the relationship between ethnicity and gender [15], [19], the relationship between ethnicity and age [24], [38], and the relationship between gender and age [23], [24], [38], [46], [48], [56] in their recognition tasks, different conclusions have been reported. For example, experimental results in [15], [19] have made different answers to the question whether gender and ethnicity are helpful in each others' recognition. The various illumination conditions and head poses in different data acquisitions, the dependency on the accuracy of the landmarks in face alignment, and especially the incomplete facial shape information in the 2D images, should account largely for this disagreement. We propose here to explore the morphological differences in the 3D shape of the face to answer the following two questions: (1) Can the 3D shape of the face reveal our gender, ethnicity and age? and (2) are the correlations between the demographic traits useful in each others' recognition task? To the best of our knowledge, this is the first work in literature that proposes a joint estimation of the demographic traits through the 3D shape differences. It is also the first work which studies the problem of age estimation from 3D faces [61].
Our approach consists of a 3D feature extraction step followed by a classification or regression step. The 3D face scans are first pre-processed to extract the facial region, then a collection of radial curves emanating from the nose tip are extracted to represent the face. Following four different pairwise curve comparison strategies, we compute four types of 3D features, for which each reflects a specific perspective in face perception. In the classification/regression step, we present the features to Random Forest in its classification mode for gender or ethnicity classification, and in regression mode for age estimation. To enhance the performance, we have also included two additional steps in our approach, the Feature Selection and the Fusion. The feature selection method is used for selecting a salient subset of features which contains the information of gender, ethnicity and age. The fusion method merges all the information from the four features by concatenation. The main contributions of this work are the following:
- -
We propose a set of 3D facial descriptors grounding in Shape Analysis of facial radial curves, with which we demonstrate that 3D shapes of our faces can reveal our gender, ethnicity and age information. These descriptions are designed to capture in different ways the morphological difference among the demographic groups.
- -
We demonstrate that gender, ethnicity and age are correlated in the 3D face, and their correlations are helpful in each others' recognition. Our conclusion is significantly different than [15], which claims that gender and ethnicity are not helpful for each others recognition.
- -
This is the first work in the literature which investigates the problem of age estimation from 3D shapes and perform joint estimation of gender, ethnicity and age in 3D. Extensive evaluations on the challenging FRGC dataset demonstrate the effectiveness of the proposed method and its robustness to facial expression variations.
The rest of the paper is organized as follows. In Section 2, the computational strategies of the 3D facial morphological features are detailed, as well as their relationship with the demographic groups. Section 3 explains the machine learning techniques adopted in our work for facial soft-biometric traits recognition. Experiments and discussions are issued in Section 4. In the end, Section 5 makes the conclusions and states some future directions.
Section snippets
Geometrical 3D features extraction
In this section, we describe four different and complementary morphological face descriptions extracted from the 3D face. These descriptions are densely computed based on shape analysis of 3D radial curves of the face. Earlier studies on 3D face recognition [13] and 4D expression recognition [7] have demonstrated the effectiveness of the proposed geometrical framework in comparing 3D faces and capturing shape deformations. We point out that although in the common geometrical shape analysis
Facial Soft-biometric traits recognition
We consider now the remaining steps in our facial demographic traits recognition algorithms. First, we perform a correlation-based Feature Selection method on our descriptions. Then, we present the selected features to Random Forest for facial classification/regression. We note that these two steps are common in many recognition problems, however, we shall present here in-deep analysis on the correlation between the facial demographic traits and the extracted 3D features. In particular, we
Experimental results
In the following experiments, we use the Random Forest method [8] in classification mode for gender and ethnicity classification, and the regression mode for age estimation. For the experiments in Sections 4.1–4.3, they are carried out on the Face Recognition Grand Challenge 2.0 (FRGCv2) dataset [47]. The FRGCv2 dataset was collected by researchers from the University of Notre Dame and contains 4007 3D face scans of 466 subjects with differences in gender, ethnicity, age and expression. For
Conclusions
This paper presents a set of new morphology-driven features extracted from the 3D shape of the face and investigates the joint demographic (gender, ethnicity and age) estimation based on them. It provides also a comprehensive study on their relevance and highlights the most informative areas of the 3D face for age, gender and ethnicity. The proposed 3D features are used individually (and fused) to perform in the same time gender, ethnicity and age estimation which makes them applicable for the
Acknowledgments
This work was supported by the ANR project 2010 INTB 0301 01, the CMCU project number 34882WK, the Ph.D. scholarship from the Chinese Scholarship Council (CSC) to Baiqiang Xia and partially supported by the FUI project MAGNUM 2. The authors would like to thank the colleagues at the Media Integration and Communication Center (MICC) of the University Florence for providing the FU-3D-Faces dataset.
References (65)
A morphogenetic analysis of facial growth
Am. J. Orthod.
(1966)- et al.
Local circular patterns for multi-modal facial gender and ethnicity classification
Image Vis. Comput.
(2014) - et al.
Computational methods for modeling facial aging: a survey
J. Vis. Lang. Comput.
(2009) - et al.
Facial gender classification using shape-from-shading
Image Vision Comput.
(2010) - et al.
Three-dimensional caricatures of human heads: distinctiveness and the perception of facial age
PERCEPTION
(1997) - et al.
The effect of ethnicity on 2D and 3D frontomaxillary facial angle measurement in the first trimester
Obstet. Gynecol. Int.
(2013) - et al.
Analysis of sexual dimorphism in human face
J. Vis. Commun. Image Represent.
(2007) - et al.
The florence 2D/31 hybrid face dataset
- et al.
Boosting 3-D geometric features for efficient face recognition and gender classification
IEEE Trans. Inf. Forensics Secur.
(2012) - et al.
Is face distinctiveness gender based?
J. Exp. Psychol. Hum. Percept. Perform.
(2006)
4D facial expression recognition by learning geometric deformations
IEEE, Trans. Cybernetics
Random forests
Mach. Learn.
Sex discrimination: how do we tell the difference between male and female faces?
Perception
The relationship between age and facial asymmetry
What else does your biometric data reveal? a survey on soft biometrics
IEEE Trans. Inf. Forensics Secur.
Facial aging, attractiveness, and distinctiveness
PERCEPTION
3D face recognition under expressions, occlusions, and pose variations
IEEE, Trans. Pattern Anal. Mach. Intell.
Demographic classification: do gender and ethnicity affect each other?
International anthropometric study of facial morphology in various ethnic groups/races
J. Craniofac. Surg.
Learning race from face: a survey
IEEE Trans. Pattern Anal. Mach. Intell.
Age synthesis and estimation via faces: a survey
IEEE Trans. Pattern Anal. Mach. Intell.
Face gender classification on consumer images in a multiethnic environment
Automatic age estimation based on facial aging patterns
IEEE, Trans. Pattern Anal. Mach. Intell.
Biologically significant facial landmarks: how significant are they for gender classification?
Human age estimation and sex classification
Image-based human age estimation by manifold learning and locally adjusted robust regression
IEEE Trans. Image Process.
Human age estimation: what is the influence across race and gender?
Correlation-based Feature Subset Selection for Machine Learning
Demographic estimation from face images: human vs. machine performance
IEEE, Trans. Pattern Anal. Mach. Intell.
Gender classification based on 3D face geometry features using SVM
Age and gender estimation based on wrinkle texture and color of facial images
Subjective experiments on gender and ethnicity recognition from different face representations
Cited by (13)
Ethnicity classification by the 3D Discrete Landmarks Model measure in Kendall shape space
2020, Pattern Recognition LettersCitation Excerpt :As a result, 3D facial data retain the full spatial information. Geometric features include mesh features [16], profiles and curvature [17], radial and isolevel curves [18,19,21], 3D landmarks-based geodesic distance [22], symmetry properties [20] and correspondence vectors [23]. Using geometric features in classification tasks was robust to texture variations.
3D Nose shape net for human gender and ethnicity classification
2019, Pattern Recognition LettersCitation Excerpt :Xia proposed a gender classification method based on 3D Facial Symmetry analysis [19]. He also combined averaged face information for gender classification [21], and extended the methods to gender facial classification tasks [25]. The advantages of the methods are based on 3D geometric feature analysis.
Coordinate-wise monotonic transformations enable privacy-preserving age estimation with 3D face point cloud
2024, Science China Life SciencesGender and ethnicity classification of the 3D nose region based on scaling invariant harmonic wave kernel signature
2023, Multimedia Tools and ApplicationsMachine-based Stereotypes: How Machine Learning Algorithms Evaluate Ethnicity from Face Data
2023, ACM International Conference Proceeding SeriesOn 3d Face attributes analysis using deep learning: a preliminary case study on gender and ethnicity recognition
2020, ACM International Conference Proceeding Series
- ☆
This paper has been recommended for acceptance by Arun Ross.
- 1
Part of this work has been published in the International Conference on Computer Vision Theory and Applications 2014 [61] and won the Best Paper Award in the area of Image and Video Understanding.