Elsevier

Neurocomputing

Volume 266, 29 November 2017, Pages 465-476
Neurocomputing

Illumination normalization based on correction of large-scale components for face recognition

https://doi.org/10.1016/j.neucom.2017.05.055Get rights and content

Abstract

A face image could be decomposed into two components of large- and small-scale components, which carry low- and high-frequency contents of the original image, respectively. The illumination field mainly locates in the spectrum of large-scale components, whereas the small-scale components hold the detailed image cues, like edge, corner, etc., which are less sensitive to the illumination changes. In this paper, we proposed a new illumination normalization framework with the idea of Correction on Large-scale Components (CLC). The logarithmic total variation (LTV) technique is firstly applied to decompose the large- and small- scale components of face images. We assume that there are two main contents in the large-scale components: the smoothly varied illumination field and the large-scale intrinsic facial features. Based on this assumption, an energy minimization framework is proposed to estimate and remove the smoothly varied field of the large-scale components in an interleaving fashion. The final normalization results can then be achieved with the integration of the smoothed small-scale components and the corrected large-scale components. Experiments on CMU-PIE, Extended Yale B and CAS-PEAL-R1 databases show that the proposed method can present a very good visual quality even on the images illuminated with deep shadow and high brightness regions, and attain promising illumination normalization results for better face recognition performance.

Introduction

Many promising face recognition algorithms remain to be highly constrained by the illumination conditions and therefore the illumination normalization is an important but challenging research issue [1], to improve the face recognition performance. Various lighting effects such as shadows, underexposure, and overexposure may cause the image either too dark or too bright, and then bias the face recognition algorithm. Sometimes, the intra-personal difference of face appearance caused by the lighting variation can even be larger than inter-personal difference. Existing face recognition methods, such as principal component analysis (PCA) [2], independent component analysis (ICA) [3], linear discriminant analysis (LAD) [4], and sparse representation [5], are sensitive to illumination variations to some extent.

In the past decades, numerous methods [6], [7], [8], [9], [10], [11], [12] have been proposed to handle lighting variations as the preprocessing step of face recognition. These methods could be roughly divided into three categories [13]: (1) modeling of illumination variations, (2) restoring the illumination to a canonical form, (3) extracting illumination invariant features. Approaches in the first category majorly utilize the learnt knowledge from the training samples to withstand possible illumination variations. In [14], Georghiades et al. observed that the set of images of the same subject with fixed pose, but under all possible illumination conditions, could conform a convex cone in the space of images. Based on this observation, a small number of training images of each face with different lighting directing were then used to reconstruct the albedo and shape of the face, similar idea can be found in [15]. In [16], Basri and Jacobs suggested that the images of the same face under varying illumination or expressions span a low-dimensional linear subspace, and hence can be represented with spherical harmonic model. Kumar et al. [17] demonstrated that realistic relighting can be effectively used to enhance face recognition accuracy via gallery set augmentation. Given a single image with near frontal illumination, several images under novel illuminations were synthesized to augment the gallery for a face classifier. Although this type of methods can model the illumination variations quite well, the main weakness is that a large number of samples, which cover various lighting conditions, are required for training. The cost of the preparation and collection of face images with various lighting conditions can be relatively high.

The methods of the second category mostly tried to restore the face images to normal illumination condition with a compensating scheme. Early algorithms focused on the simple gray-scale adjustment on the face image, e.g., Gamma correction, histogram equalization (HE) [18], histogram match (HM) [19], etc. However, these methods simply adjust the gray level distribution and ignore the physical illumination [10]. Later methods of this category were based on various techniques like morphing face [17], quotient image [20], low-frequency domain [21], etc. Reasonably good results can be achieved with these methods, yet a strict alignment of the image toward a fixed shape model seems to be always required. Unfortunately, image alignment can be also affected with varying illuminations [22].

Approaches of the third category are the earliest but most widely used ones. In the earlier literatures, researchers proposed to use simple descriptors, e.g., logarithmic transformation, edge map, LBP, etc., for face recognition. These methods are easy to implement but the space of improvement on face recognition performance with these methods may be limited. Other methods tackled the lighting variation issues based on the concept of time-frequency analysis, by seeking face features that are insensitive to various illumination conditions. Most of these methods were based on the Lambertian reflectance model [16], which assume the albedos of salient facial parts shall be high frequency contents, i.e., the small-scale components of the images, whereas the illumination parts correspond to the low frequency contents (the large-scale components) [19], [23]. Jobson et al. [24] proposed the Multiscale Retinex (MSR) method to reduce the effect of illumination by dividing the facial image with a smoothed image. Self-quotient image (SQI) [25] technique was developed subsequently to compute more reasonable smoothed image with the weighted Gaussian filter. Inspired by the SQI technique, Chen et al. [23] proposed the logarithmic total variation (LTV) method to attain better edge-preserving capability with the total variation model. In [26], Zhang et al. transformed image into the gradient domain firstly and extracted illumination insensitive features, called Gradientfaces, which was shown to be robust to different degree of lighting and noise. Although promising performance was reported, these aforementioned methods only focused on the analysis of small-scale components but omitted the large-scale components, which may also carry useful image cues for face recognition. Meanwhile, a normalized illumination, i.e., frontal-illumination image with a good visual quality can be certainly reconstructed with the consideration of both large- and small-scale components.

Other than the three categories of methods, there are several works, which proposed to fuse the local and holistic features to address the illumination variation issue for face recognition [22], [27], [28], [29]. Particularly in [22], Xie et al. proposed a new illumination normalization method that merges the features from both small- and large-scale components. It was shown that their normalized images can preserve more facial details for better boosting of face recognition performance. However, the work in [22] may not be able to properly tackle the images with extreme lighting condition, and hence hinder the effectiveness of many image processing methods and steps.

A common assumption for illumination normalization is that the illumination changes smoothly in the scene, while the edges remain similar between the original image and the normalized one [30]. However, as we will discuss later, this smoothly properties are inescapable affected by the small-scale features if normalize directly on the original scene. In this paper, we decompose images into the large- and small-scale components firstly. We argue that there are mainly two sub-component in the large-scale components, i.e., the illumination field and the large-scale intrinsic facial features. We then use two mathematical representations to approximate these two sub-components for the systematically systematic decomposition of the large-scale components. We formulate the estimation and removal of the illumination field from the large-scale components in an energy minimization framework, which is denoted as correction operation throughout this paper. Afterward, the final normalized result can be obtained with the integration of the small-scale features and the corrected large-scale features. For simplification, the proposed method is termed as Correction of Large-scale Components (CLC). After applying CLC to images under various illumination conditions, the normalized images are used for face recognition on three face databases: CMU-PIE, Extended Yale B and CAS-PEAL-R1.

Section snippets

Related works and motivation

Based on the Lambertian model [16], a face image could be describe by I(x,y)=R(x,y)L(x,y)where I is the intensity of the observed face image at pixel (x, y), R is the reflectance (albedo), and L is the illumination effect. Here, the nature of L is determined by the illumination source, whereas R is determined by the characteristics of the image objects. However, it is an ill-posed problem to estimate R from I [31]. To make the problem solvable, Chen et al. proposed a practical methodology [23],

The proposed framework

In this section, we will present the formulation of the proposed method for illumination normalization based on decomposing the large-scale components into two multiplicative components. An energy minimization method is proposed to obtain these two multiplicative components. The overview and each step of the formulation are described in the following sections.

Experimental setting

In this section, we first show the normalization results of CLC on three illumination-varied face database: CMU-PIE [44], Extended Yale B [14] and CAS-PEAL-R1 [45]. Then we present the results of illumination normalization and quantitative evaluation between CLC and some other popular face illumination normalization methods. The face recognition results of using different normalization algorithms are also reported to evaluate how the performance of the proposed method is improved compared with

Conclusion

A novel technique for face illumination normalization has been proposed in this paper. Rather than performing the illumination normalization directly on the original face images, we decompose the original image into two components, the large-scale components and the small-scale components. A energy minimization framework is then proposed, with the aim of removing illumination field from the large-scale components. The final normalization result could be achieved by reconstruction of the

Acknowledgment

This work was supported partly by the Fundamental Research Funds for the Central Universities of China (No. A03017023701112), and the 111 project (B14039). We would like to thank the authors of [22] who have offered the code of R&S (LOG-DCT) and R&S (NPL-QI).

Xiaoguang Tu is a Ph.D. candidate with the School of Communication and Information Engineering at University of Electronic Science and Technology of China (UESTC). His research interests include convex optimization, computer vision and deep learning.

References (53)

  • J. Wright et al.

    Robust face recognition via sparse representation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2009)
  • WuZ. et al.

    One-shot deep neural network for pose and illumination normalization face recognition

    Proceedings of IEEE International Conference on Multimedia and Expo

    (2016)
  • LiuZ. et al.

    Face recognition under varying illumination

    Neural Comput. Appl.

    (2013)
  • TanX. et al.

    Enhanced local texture feature sets for face recognition under difficult lighting conditions

    IEEE Trans Image Process.

    (2010)
  • WangB. et al.

    Illumination normalization based on weber’s law with application to face recognition

    IEEE Signal Process. Lett.

    (2011)
  • Juefei-XuF. et al.

    Encoding and decoding local binary patterns for harsh face illumination normalization

    Proceedings of 2015 IEEE International Conference on Image Processing (ICIP)

    (2015)
  • LeeP.-H. et al.

    Illumination compensation using oriented local histogram equalization and its application to face recognition

    IEEE Trans. Image Process.

    (2012)
  • F.R. Al-Osaimi et al.

    Illumination normalization of facial images by reversing the process of image formation

    Mach. Vis. Appl.

    (2011)
  • A.S. Georghiades et al.

    From few to many: illumination cone models for face recognition under variable lighting and pose

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2001)
  • LiuZ. et al.

    A novel classification method for palmprint recognition based on reconstruction error and normalized distance

    Appl. Intell.

    (2013)
  • R. Basri et al.

    Lambertian reflectance and linear subspaces

    IEEE Trans Pattern Anal. Mach. Intell.

    (2003)
  • R. Kumar, M. Jones, T.K. Marks, Morphable Reflectance Fields for Enhancing Face Recognition, 2010, pp....
  • A. Shashua et al.

    The quotient image: class-based re-rendering and recognition with varying illuminations

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2001)
  • ChenW. et al.

    Illumination compensation and normalization for robust face recognition using discrete cosine transform in logarithm domain

    IEEE Trans. Syst. Man Cybern. Part B Cybern.

    (2006)
  • XieX. et al.

    Normalization of face illumination based on large-and small-scale features

    IEEE Trans. Image Process

    (2011)
  • ChenT. et al.

    Total variation models for variable lighting face recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2006)
  • Cited by (0)

    Xiaoguang Tu is a Ph.D. candidate with the School of Communication and Information Engineering at University of Electronic Science and Technology of China (UESTC). His research interests include convex optimization, computer vision and deep learning.

    Jingjing Gao is with the School of Electronic Engineering at University of Electronic Science and Technology of China. She received her Ph.D. degree from UESTC in 2014. Her research interest is machine learning and pattern recognition.

    Mei Xie is a professor with School of Electronic Engineering at University of Electronic Science and Technology of China (UESTC). She received the Ph.D. degree in signal and information processing (SIP) from UESTC in 1997. Between 1997 and 1999, she studied in University of Hong Kong and University of Texas for the postdoctoral degree, respectively. Her research interests include pattern recognition, computer vision and artificial intelligence.

    Jin Qi is with the School of Electronic Engineering at University of Electronic Science and Technology of China. He received his Ph.D. degree from the Institute of Automation, Chinese Academy of Sciences in 2005. His research interest is computer vision and machine learning.

    Zheng Ma is a professor with the School of Communication and Information Engineering at University of Electronic Science and Technology of China (UESTC). His research interests include convex optimization, computer vision and image processing.

    View full text