Long-Term Face Image Analysis Based on Canonical Correlation Analysis on Physical and Psychological Evaluation of Face

Abstract

In this paper, we analyze the relationship between impression of facial skin and skin pigmentation distribution by applying Canonical Coefficient Analysis (CCA) to multiple physical and psychological features obtained from facial skin. Based on the acquired relationship expression, we modulate the skin pigment features, and appearances of the face with arbitrary psychological features are reproduced. In our previous work, we applied principal component analysis (PCA) to the melanin pigment variation of the facial skin, and we obtained individual differences in it occurring over seven years. In the previous method, as the factor causing individual difference, we considered the frequency of UV care. However, actual skin appearance is thought to depend not only on melanin but also on several other factors. Therefore, in this study, we photographed the faces of women of various ages for 12 years, and at the same time obtained psychological features of appearance. As physical features, melanin and hemoglobin pigmentation distributions, shading and the frequency of UV care for 12 years were obtained. Subjective evaluation values were acquired as psychological features. As a result of CCA, it was found that the whole face can be made lighter in appearance by using UV protection every day continuously for six years or more.

A part of this paper will be published in Journal of Imaging Science and Technology.

1 Introduction

Human face is the most watched part in the human body. We obtain many information from face, which are divided into two kinds of feature values. One is called “physical features” such as skin condition or facial structure, and the other is called “psychological features” such as health condition or appearance of age. Facial appearance depends on these two features largely.

People, especially, women have a strong interest in their appearance of face or skin. In the beauty industry, therefore, many kinds of cosmetics have been developed for improving appearance, and it is expected to practically predict the effect of cosmetics. For example, we are able to simulate face with makeup by web applications [1]. This system enable to predict the effect of cosmetics anytime, anywhere at low cost and promote sales of cosmetics.

Moreover, there is a lot of research on simulation of facial appearance in recent years. For example, PCA makes it easier to obtain feature values. Lantis et al. provided a framework for simulation of aging effects on facial image. By applying PCA to facial landmarks, they simulated facial structure in any age based on classification of age [2]. Suo et al. also predicted appearance of face for the long period by changing parts of face for the short period based on result of applying PCA to facial image database divided by parts or ages [3]. However, individual differences were not considered in this method.

As described above, by using PCA, we are able to obtain facial feature values relatively easily from information such as facial structure and skin texture. Most of these researches directly analyze grayscale or RGB images. However, RGB color images don’t consider layer structures for skin properly, because color characteristics change depending on the device. For this reason, it is thought that we can analyze for face or skin more effectively by taking into account melanin and hemoglobin colors which is main components of skin color. Tsumura et al. proposed the technique to extract pigmentation distribution of melanin and hemoglobin from a single skin color image by applying independent component analysis (ICA) [4, 5]. Melanin and hemoglobin color can be obtained regardless of light sources or characteristics of camera by ICA in their method.

Toyota et al. analyzed pigmentation distribution in whole face. Toyota et al. obtained feature values of skin pigmentation in whole face by ICA and PCA and simulated appearance of face having arbitrary psychological features [6]. This method can perform to synthesize appearance of face considering the changes of the age. However, as a result of a subjective evaluation experiment by experts, there was large difference between age of synthesized images and evaluated results. For this reason, Hirose et al. analyze variation of facial landmarks representing facial structure and surface reflection component representing wrinkles and pores in addition to skin pigmentation distribution by PCA and Multiple Regression Analysis (MRA) [7]. They succeeded to reduce the age difference of synthesized image and real images. Since this simulation is based on changing averaged features in the database with the same age, each synthesized image lost the individual characters. However, the actual aging depends on individuals, and it is expected to predict the appearance of face by considering the individual characters. Therefore, in a previous study, the frequency of UV protection is considered as a factor that causes individual differences in skin color change. [8]. PCA was applied to melanin pigment components extracted from face image changes of same people for 7 years. As a result of comparing melanin principal component score changes of each person, it was found that whole facial skin and skin around cheeks of those who don’t use UV protection tend to increase melanin pigment compared with those who use it every day for 7 years. However, it seems that the appearance of the skin is influenced not only by melanin but also by several factors.

In this paper, therefore, CCA is applied to multiple physical and psychological features, and the relationship between skin appearance and facial skin pigment is analyzed. Specifically, first of all, CCA is applied to physical features. In a previous study [8], only melanin pigment was analyzed, but in this study, we perform CCA on the multiple skin pigmentary changes for 12 years and the frequency of UV protection for 12 years. By doing this, we analyze the influence of frequency of UV protection of each year on skin pigment changes for 12 years. Next, in order to realize face simulator, CCA is performed on features of skin pigment which are physical features and the subjective evaluation value of skin which are psychological features. Thus, a relational expression between physical features and psychological features can be derived. By modulating the physical features based on this relational expression, reproductions of facial appearance in arbitrary psychological features are performed.

2 Construction of Facial Image Database

The database is consisted of facial images, real age, and the frequency of UV protection. First of all, Japanese women whose age were from 10 s to 80 s are photographed in 2003 and 2015. The number of subjects was 86 in 2003 and 161 in 2015, in total, 247 facial images. Note that 60 women were the same person. Breakdown of the number of subjects and distribution of age in the database are shown in Fig. 1.

Fig. 1.

Breakdown and Distribution of age of the database: (a) Breakdown of the number of subjects, (b) Distribution of age in the database

These photographs were taken in imaging system shown in Fig. 2. This imaging system was surrounded by blackout curtains in order to eliminate the effect of ambient light. As the light source, there were four fluorescent lights so that the lights surrounded the camera as shown in Fig. 2.

Fig. 2.

Overview of imaging system

The cameras, NikonD1 and NikonD2H were used; the former was used in 2003 and the latter in 2015. In order to prevent shaking of face, we used the support for neck and head which was fixed on backrest of chair. We obtained facial image without specular reflectance by arranging polarization filters in front of the camera and the light sources mutually perpendicularly. There was difference in color tone between images taken in 2003 and those in 2015 due to using different camera. For this reason, we matched color tone of images taken in 2003 with those in 2015 by MRA.

Figure 3 shows a sample of the captured facial image. These captured facial images were required to be normalized in order to remove influence caused by variation of individual facial shapes on applying PCA accurately to images later.

Fig. 3.

Sample of captured image

For this reason, we used FUTON (Foolproof UTilities for facial image manipulatiON system), which was facial image synthesis system developed by Mukaida et al. [9]. First, we obtained facial landmarks representing facial structure and extracted facial areas from captured facial images. Second, we morphed shape of facial images into an image of an average face which was made from facial images in database. As a result, we obtained normalized facial images while keeping individual skin texture information. The overview of this process is shown in Fig. 4.

Fig. 4.

Overview of normalization process for facial images (Each number in the lower right shows the number of images; the average face image is only one).

3 Acquisition of Features

This section shows how to acquire both physical and psychological features. In this paper, frequency of UV protection and skin pigmentation distributions are defined as physical features. While a subjective evaluation value for a skin image described later is defined as a psychological features.

3.1 Frequency of UV Protection

Frequency of UV protection was obtained by three-tiered evaluation (1: Never, 2: Sometimes, 3: Daily) in winter of 2003 and 2010, and six-tiered evaluation (1: Daily in the last five years, 2: Daily in summer for the last five years, not in winter, 3: Used period is longer than unused period in the last five years, 4: Used period is shorter than unused period in the last five years, 5: Rarely in the last five years, 6: Others) in 2015. Figure 5 shows age distribution of frequency of UV protection in database.

Fig. 5.

Age distribution of use frequency of UV protection: (a) in 2003, (b) in 2015 (Regrouped), (c) in 2015 (Original)

3.2 Skin Pigmentation Distribution

3.2.1 Extraction of Skin Pigmentation by ICA

Skin structure can be broadly divided into epidermis where melanin pigment exist and dermis where hemoglobin pigment exist. Assuming that skin color depends on these 2 kinds of pigmentation density, the skin color vector can be represented by three vectors as shown in Fig. 6: melanin, hemoglobin, and shading, by using ICA and modified Lambert-Beer’s rule [4, 5].

Fig. 6.

Overview of independent component analysis

Figure 7(a) and (b) show a sample of the extracted melanin and hemoglobin pigmentations, and (c) shows the shading in the whole facial image. As you can see in Fig. 7(a), the mole and pigmented spot can be obtained as melanin component. The redness caused by pimples can be seen in Fig. 7(b), and the shadow caused by uneven facial features can be recognized in Fig. 7(c). In this study, we obtained melanin and hemoglobin pigmentation and shading components of facial skin by this method.

Fig. 7.

The results of independent component analysis for extraction of pigmentation components: (a) melanin, (b) hemoglobin, (c) shading

3.2.2 Acquiring Skin Pigmentation Features by PCA

We obtained feature values of uneven pigmentations by applying PCA to melanin, hemoglobin, shading components of 247 facial images extracted in Sect. 3.2.1. PCA is a statistical method to grasp a tendency and features of data by multivariate analysis. This analysis calculates the linear sum of each variable in data group constructed from any variable, and defines a new index as the first component. The second component is defined in such a way that is perpendicular to the first component, and other components are defined similarly. The n-dimensional l-th vector in dataset can be represented as the approximated vector as follows:

$$ \hat{x}_{i} \left( {x_{i,1} ,x_{i,2} , \cdots ,x_{i,n} } \right) = \sum\nolimits_{m = 1}^{M} {w_{i,m} P_{m} } $$

(1)

where n is the total number of pixels; 512 × 512. $x_{i}$ means an image represents each pixel as $x_{i,1} ,x_{i,2} , \cdots ,x_{i,n} ,$ as shown in Fig. 8. M is the total number of principal components, and the m-th principal component vector is defined as $P_{m}$. The principal component score $w_{i,m}$ is the weight of the $P_{m}$, in the image $\hat{x}_{i}$.

Fig. 8.

Overview of principal component analysis (n = number of pixels in the image; other variables as defined in the text).

We applied PCA to melanin, hemoglobin, shading components whose size were 512 × 512 pixels. In this paper, we regarded one pixel as one variable. That is, 247 women’s melanin, hemoglobin, shading components existed in 512 × 512-dimensional space respectively. As a result of PCA, we obtained 246 principal components for each skin pigmentation components.

For example, if you compare the principal component scores of the same person between 2003 and 2015, you can obtain how the skin pigment components of that person changed in 12 years. Also, by comparing the principal component scores with different people, individual differences in skin pigment components can be obtained. In this study, principal component score were used as physical features representing skin pigment.

3.3 Subjective Evaluation Value of Skin

Subjective evaluation experiments of skin were performed on face images taken in 2015. Three experts determined the evaluation value of the total of 8 items: a kind of spots (on the whole face), 4 kinds of wrinkles (on forehead, corners of the eyes, under the eyes, and nasolabial grooves), and 3 kinds of sagging (on whole face, under the eyes, and corners of mouth). They evaluated the above 8 items in 4 grades for each facial photograph, compared to a photo scale showing stepwise the degree of wrinkles and sagging. In this study, each average values of evaluation values were used for analysis as psychological features.

4 CCA for Multiple Features

In this section, we explain the method of CCA and the results of CCA on physical and psychological features.

4.1 The Method of CCA

CCA is a method of analyzing the relationship between multivariables by combining variables so that the correlation coefficients between the two variable groups is high. Let us consider the case where there are variable group X = {X₁, …, X_p} and variable group Y = {Y₁, …, Y_q}, (p < q). The canonical coefficients A and B which are weights are set and synthesized so that the correlation coefficient R of the weighted linear sum U of the variable group X and the weighted linear sum V of the variable group Y becomes maximum. Interpretation of a canonical variable is performed by analyzing canonical loading which is a correlation coefficient between each variable constituting a variable group and a canonical variable. The overview of canonical correlation analysis is shown in Fig. 9.

Fig. 9.

Overview of canonical correlation analysis

4.2 CCA on Physical Features

In order to analyze the relationship between multiple skin pigments and frequency of UV protection in each year, CCA was performed among physical features. Specifically, the differences between the principal component scores of 2003 and 2015 of 1st~8th principal components of melanin and 1st~15th principal components of hemoglobin and shading, total of 38 variables were set as the variable group X. As the variable group Y, frequency of UV protection of 2003, 2010, and 2015, total of 3 variables were set. As a result of CCA on these, the 1st canonical variables V₁ and U₁ became significant. Here, V₁ represents the 1st canonical variable of the variable group Y, and U₁ represents the 1st canonical variable of the variable group X. Tables 1 and 2 shows canonical loadings of V₁, U₁.

Table 1. Canonical loading of V₁

Table 2. Canonical loading of U₁

The PC in Table 2 represents the principal component. From Table 1, V₁ has the largest negative correlation with the frequency of UV protection of 2003, has the next largest correlation with that of 2010, and has no correlation with that of 2015. That is, V₁ is the value that increases as the person who didn’t use UV protection in the past. On the other hand, from Table 2, U₁ has positive correlation with variation of melanin pigment of whole face, cheeks, and forehead, and hemoglobin pigment of forehead and around eyes. That is, U₁ becomes larger as these components increase. Therefore, taken together, it can be said that the less frequencies of UV protection in 2003 and 2015 are, the more increase the amounts of melanin in the whole face, cheeks, and forehead, and hemoglobin of forehead and around eyes in 12 years. Actual 12-year changes of normalized face images whose V₁ are smallest and largest are shown in Fig. 10.

Fig. 10.

Actual 12-year changes of normalized face images whose V₁ is: (a) smallest (taken in 2003), (b) smallest (taken in 2015), (c) largest (taken in 2003), (d) largest (taken in 2015)

From Fig. 10(a) and (b), those who had a high frequency of UV protection in 2003 and 2010 could brightened the whole face 12 years later. In contrast, from Fig. 10(c) and (d), those who had low frequency of UV protection in 2003 and 2010, as you can see, the whole face and cheeks had changed darkly in 12 years. Therefore, it seems necessary to use UV protection everyday for more than 6 years in order to keep beautiful skin.

4.3 CCA on Physical and Psychological Features

In order to reproduce the appearance of face, CCA was performed on psychological features and physical features. Specifically, subjective evaluation values for 1 spot, 4 wrinkles, 3 sags, 8 variables in total were set as variable group Y. As variable group X, principal component scores in 2015 of 1st~8th principal components of melanin and the 1st~15th principal components of hemoglobin and shading, total of 38 variables were set. As a result of CCA on these, 1st, 2nd and 3rd canonical variables became significant. The canonical correlation coefficients of 1st, 2nd and 3rd canonical variables were 0.93, 0.67 and 0.60. Let V be canonical variable of variable group Y, and U canonical variable of variable group X. Tables 3 and 4 show particularly large canonical loadings of the 1st canonical variables V₁ and U₁. From Table 3, V₁ has a high positive correlation with spots, wrinkles, and sagging. That is, the more conspicuous a person’s spots, wrinkles and sag in her facial skin are, the larger V₁ is. On the other hand, from Table 4, U₁ has a positive correlation with melanin pigment of cheeks and whole face, and shading of the nasolabial folds. That is, the stronger these pigment components are, the larger U₁ is. Canonical correlation coefficient V₁ and U₁ is 0.93. Therefore, it can be said that melanin of cheeks and shading of whole face and nasolabial lines are darker for people whose spots, wrinkles and sags are conspicuous.

Table 3. Canonical loading of V₁

Table 4. Canonical loading of U₁

Next, Tables 5 and 6 show particularly large canonical loadings of the 2nd canonical variables V₂ and U₂. From Table 5, V₂ has a positive correlation with spots on whole face and a negative correlation with sagging on mouth corners. Thus, the more conspicuous a person’s spots on whole face are but the less conspicuous a person’s sagging of mouth corners are, the larger V₂ is, while, the more conspicuous a person’s sagging of mouth corners are but the less conspicuous a person’s spots on whole face are, the smaller V₂ is. On the other hand, from Table 6, U₂ has a positive correlation with melanin around eyes, nose ridge and cheeks, and hemoglobin of nose and cheeks. U₂ also has a negative correlation with melanin in whole face. This means the stronger melanin pigment around eyes, nose ridge and cheeks, and hemoglobin of nose and cheeks are, the larger U₂ is, while the stronger melanin pigment in whole face is, the smaller U₂ is. Canonical correlation coefficient of V₂ and U₂ is 0.67. Therefore, it can be said that people with conspicuous spots have strong melanin around eyes, nose ridge and cheeks, and hemoglobin of nose and cheeks, while people with conspicuous sagging on mouth corners have strong melanin in whole face.

Table 5. Canonical loading of V₂

Table 6. Canonical loading of U₂

Similarly, let us consider the 3rd canonical variables. Tables 7 and 8 show particularly large canonical loadings of the 3rd canonical variables V₃ and U₃. As shown in Table 7, V₃ has a positive correlation with wrinkles on corners of eyes and a negative correlation with sag on mouth corners. On the other hand, from Table 8, U₃ has a positive correlation with hemoglobin around nose and cheeks, and shading around eyebrows and eyes, and a negative correlation with melanin of nose ridge. Canonical correlation coefficient of V₃ and U₃ is 0.60. Therefore, it can be said that people with conspicuous winkles on corners of the eyes have red cheeks and nose and strong shading eyebrows and eyes, while people with conspicuous sagging on mouth corners have strong melanin of nose ridge. In this way, by CCA on physical features and psychological features, skin pigment distribution corresponding to the subjective evaluation value of the skin could be acquired.

Table 7. Canonical loading of V₃

Table 8. Canonical loading of U₃

5 Reproduction of Face in Arbitrary Psychological Features

From the results of CCA shown in Sect. 4.3, the following relational expression is established.

$$ \left( {\varvec{A}^{\text{T}} } \right)^{ + } \cdot diag\left( {\left[ {R_{1} ,R_{2} , \cdots ,R_{7} ,R_{8} } \right]} \right) \cdot \varvec{V}^{\text{T}} = \varvec{X}^{\text{T}} $$

(2)

Here, A represents canonical coefficient for variable group X, R represents canonical correlation coefficient, V represents canonical variable of variable group Y, diag represents a diagonal matrix, + represents a pseudo inverse matrix, and T represents a transposed matrix. From Eq. (2), we obtained principal component score X, and reproduced appearances of face in arbitrary V₁, V₂, V₃. The results reproduced so that V₁, V₂, and V₃ are minimum and maximum are shown in Fig. 11. Reproduced images are emphasized by doubling the actual principal component score change in order to make them easy to see. In Fig. 11(a), color unevenness of the whole face is small, but in Fig. 11(b), cheeks are dark, shadows of the corners of mouth and wrinkles of the corners of the eyes are conspicuous. Also, in Fig. 11(c), cheeks are whitish and there are sag in the corners of mouth, whereas in Fig. 11(d), cheeks are dark and the sagging of the corners of mouth are less noticeable than in Fig. 11(c). From Fig. 11(e), shading of corners of the mouth is dark. While Fig. 11(f) shows darker eyebrows and stronger wrinkles of corner of the eyes than that of Fig. 11(e). From the above, as a result of modulating the pigment distribution based on the results of CCA, we could reproduce the appearance of the skin almost similar to the examination.

Fig. 11.

The results of the appearance of a face in arbitrary psychological features: (a) V₁ is smallest, (b) V₁ is largest, (c) V₂ is smallest, (d) V₂ is largest, (e) V₃ is smallest, (f) V₃ is largest

6 Conclusion

In this study, CCA was performed on multiple physical and psychological features, and the relation between skin appearance and skin pigment distribution was analyzed. As a result of CCA on physical features, skin pigment distribution affected by frequency of UV protection was acquired. From this, it is thought that beauty of the skin can be maintained by performing UV care everyday continuously for more than 6 years. In addition, as a result of CCA for physical features and psychological features, principal components of skin pigment which greatly affects the appearance of the skin was acquired. Furthermore, based on this result, physical features were modulated, and appearances of the face in arbitrary psychological features were reproduced. In the future, we will examine the estimation formula for reproducing the appearance of the face, and improve reproduction of appearance of the face.