Elsevier

Information Sciences

Volume 481, May 2019, Pages 174-188
Information Sciences

Face hallucination through differential evolution parameter map learning with facial structure prior

https://doi.org/10.1016/j.ins.2018.12.064Get rights and content

Abstract

Current learning based face hallucination approaches mainly focus on how to design a reasonable objective function, such as using different assumptions and incorporating different regularization terms, but do not give a reasonable way of selecting the model parameters. In this paper, we propose to exploit the facial structure prior to learn a parameter map based on differential evolution. Specifically, we claim that different position patches have different parameter settings because of their different statistical properties, and patches from the same position of different face images should have similar parameter settings. As a result, we first learn a parameter map for each training sample by leveraging an evolutionary algorithm based on differential evolution, and then fuse these learned parameter maps to an optimal parameter map for testing via mean-pooling strategy. Finally, we use the predicted parameter map to guide the co-occurrence relationship modeling in different regions of the input low-resolution (LR) face image. Experimental results demonstrate that, even without seeing the ground truth, results of proposed parameter map learning method are comparable to or better than those traditional unified parameter setting methods and some recently proposed deep learning methods.

Introduction

The goal of face hallucination system is to estimate a sharp high-resolution (HR) face image from a low-resolution (LR) observation [1], [22], which is a fundamental problem and a very active area [14], [15], [33], [46]. It benefits the subsequent image processing and analysis in some face analysis applications, e.g., face detection, alignment, verification, and recognition, especially when the size of the observed face image is very small [50].

The large body of work on face hallucination can be roughly divided into two classes:

  • Global model based face hallucination algorithms focus on learning the holistic prior by integrating a global parametric model, e.g., Principal Component Analysis (PCA) [3], [34], Non-negative Matrix Factorization (NMF) [25], [40], Locality Preserving Projections (LPP) [29], [49], and Canonical Correlation Analysis (CCA) [12]. The common idea of these approaches is to fit the LR testing face image to LR sample space with a linear combination coefficients, i.e., the reconstruction weights. An HR face image is then rendered by replacing the LR training face images with their HR correspondences, while retaining the same reconstruction weights obtained in the LR space.

  • By dividing a global face into small patches, these local patch based face hallucination algorithms predict the HR patch via explicit regression or implicit representation. The former directly learns the regression relationship between the LR and HR patches, and representative methods include Local Linear Transformation (LLT) [13]. Anchored Neighborhood Regression (ANR) [32], SRLSP [16], and Neural Networks (NN) [6], [37], [47]. Instead, the implicit representation methods predict the HR patch via different patch representation models, e.g., Neighbor Embedding (NE) model [4], Least Squares Representation (LSR) [28], Sparse Representation (SR) [40], Locality-constrained Representation (LcR) [18], or their variants [9], [10], [23], [24], [30], [41], [47]. The key issue of these approaches is how to obtain the optimal reconstruction weights and preserve the similar manifold structures of LR and HR couple spaces.

Both global and local approaches have different advantages and weaknesses. Super-resolution results of global based methods can well maintain the global face appearance variations, but the reconstructed face images lack detailed features. Through a patch dividing strategy, these local patch based methods have much stronger representation power, thus can well reveal the non-linear manifold structure of face image space [4], [17], [18], [19]. For example, NE based methods [4], [9], [41] assume that LR patches and its HR counterparts share similar local geometry. Ma et al. [28] developed a novel face hallucination method by performing collaboratively over all the training face patches at the same position to the input LR patch. To improve the representation ability, sparsity [10], [36], [40] and locality [9], [18], [45] priors have been used to regularize the patch representation objective function.

More recently, to reconstruct the latent HR image locally while thinking globally, Deep Neural Networks (DNNs), especially deep convolutional neural networks, have been applied to learn the mapping from the LR images to their HR counterparts and shown strong learning capability and accurate prediction of HR images [20], [31], [43].

Benefiting from the position information (or location information), the aforementioned position-patch based face hallucination algorithms can well capture the facial structure and infer a good result. However, they have the following two drawbacks. On one hand, they need the original HR image to measure the prediction errors and obtain the optimal parameter settings. But the original HR image is not always available in practice. On the other hand, they set a unified parameter experimentally (which need the reference images) for all the patches of one face image and neglect the facial structure prior. For example, in NE based methods, they set the same neighbor number K for all patches; in sparse representation or locality-constrained representation, they set the same regularization parameter for all patches. Inspired by the work [38], [42], which utilize different model parameters when reconstructing different regions, in this paper we propose a patch adaptive parameter setting algorithm. As we know, human face is a high structured object, and different regions have different textural characteristics, e.g., the face contour, eyes, and mouth have detailed texture, while the forehead and cheek are flat. Therefore, it is not reasonable to utilize a unified regularization parameter to represent all image patches of one face image.

To handle the aforementioned issues, in this paper we propose a parameter map learning method by exploring the specificity of human face. The basic assumption behind our proposed method is that (i) patches from different positions should have different parameter settings, (ii) patches from the same position of different person should have similar parameter settings. Therefore, we leverage an evolutionary algorithm based on differential evolution to learn a parameter map for each training sample. And then, we fuse these learned parameter maps to an optimal parameter map for testing via mean-pooling strategy off-line. Without seeing the original HR face image, our method can reasonably choose different parameters for different patches. Fig. 1 shows the flow diagram of the proposed parameter map learning approach. Image super-resolution is a very hot topic and becomes a test bed for many emerging models, especially recently very popular deep learning techniques. The proposed parameter learning framework is not trying to defeat all super-resolution methods [21], [39], [48]. We hope that our study can inspire the field of face hallucination a lot. Specifically, the proposed parameter learning based face hallucination method has the following contributions:

  • Compared with conventional face hallucination methods, which need the ground truth HR face images (are not always available in practice) to tediously tune their model parameters, in this paper we develop a parameter learning scheme to predict the model parameter according to the facial structure prior.

  • By learning a parameter map for each training sample, we transfer the tedious parameter tuning processes to the off-line training phase. In the on-line testing phase, we only need to use the fused optimal parameter map as a guidance.

  • We introduce an evolutionary algorithm based on differential evolution to generate the parameter map. This turns the trivial parameter tuning to differential evolution problem whose global solution can be easily reached.

The rest of this paper is organized as follows. Section 2 gives some notations used in this paper, and briefly reviews the position-patch based face hallucination methods. Section 3 details our proposed method, including differential evolution parameter map learning with structure prior and parameter map predicting via mean-pooling. Experimental results are presented in Section 4 to demonstrate the effectiveness of the proposed face hallucination method, when compared with some state-of-the-art original HR face guided parameter setting face hallucination approaches. Finally, Section 5 provides some limitations, concluding remarks and possible future work.

Section snippets

Notations

Let ItL denotes an LR observation image, the goal of face hallucination is to predict its HR version ItH by learning the relationship between the LR training samples and their HR counterparts, IL={I1L,I2L,,INL} and IH={I1H,I2H,,INH}, where N is the size of the training set. As for a patch based face hallucination method, we decompose the observed LR image, LR training face images, and HR ones into M small patches, {xt(p,q)|1pU,1qV}, {xi(p,q)|1pU,1qV}i=1N and {yi(p,q)|1pU,1qV}i=1N.

Overview of the proposed method

Instead of using a unified neighbor number K for the all patches of one face image, we try to learn a parameter map K beforehand, and then utilize it to guide the following parameter settings of the face hallucination model. Here, we use a matrix with U × V to represent the parameter map, and the element of the map is the parameter setting of the corresponding position, e.g., K(p,q) denotes the neighbor number K of position (p, q) for the input face image.

Upon acquiring the parameter map K, we

Database

In this paper, we conducted experiments on the public CAS-PEAL-R1 face Database [11]. We select 1040 neutral expression and normal illumination face images from the frontal subset. All the images are automatically aligned by of [8], [26], [27] and are cropped to 128  ×  112 pixels with the center image patches. The input LR face images are generated by Gaussian blurring (with window of 7 × 7 pixels) with standard deviation 1.6, and then downsampling by a decimation factor of 4 to produce the

Discussion, limitations and future work

Traditional local patch based methods achieve great success in the face hallucination problem, however, they need the original HR face (are not always available in practice) to guide the parameter settings and set the parameters uniformly for all the patches of one face image. In order to solve these problems, in this paper we derived a parameter setting strategy for these local patch face hallucination methods. To make the regularization model adaptive to different regions of face image, we

Acknowledgments

The research was supported by the National Natural Science Foundation of China under Grants 61501413, 61503288 and 61773295, and was also partially supported by JSPS KAKENHI Grant number 16K16058.

References (50)

  • S. Baker et al.

    Hallucinating faces

    FG

    (2000)
  • M. Bevilacqua et al.

    Low-complexity single-image super-resolution based on nonnegative neighbor embedding

    BMVC

    (2012)
  • A. Chakrabarti et al.

    Super-resolution of face images using kernel PCA-based prior

    IEEE Trans. Multimedia

    (2007)
  • H. Chang et al.

    Super-resolution through neighbor embedding

    CVPR

    (2004)
  • S. Das et al.

    Differential evolution: a survey of the state-of-the-art

    IEEE Trans. Evol. Comput.

    (2011)
  • C. Dong et al.

    Image super-resolution using deep convolutional networks

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2016)
  • D.L. Donoho et al.

    Stable recovery of sparse overcomplete representations in the presence of noise

    IEEE Trans. Inf. Theory

    (2006)
  • M. Everingham et al.

    Hello! My name is... Buffy – automatic naming of characters in TV video

    BMVC

    (2006)
  • R.A. Farrugia et al.

    Face hallucination using linear models of coupled sparse support

    IEEE Trans. Image Process.

    (2017)
  • W. Gao et al.

    The CAS-PEAL large-scale chinese face database and baseline evaluations

    IEEE Trans. Syst. Man Cybern. Part A-Syst. Hum.

    (2008)
  • H. Huang et al.

    Fast facial image super-resolution via local linear transformations for resource-limited applications

    IEEE Trans. Image Process.

    (2011)
  • J. Jiang et al.

    Noise robust position-patch based face super-resolution via Tikhonov regularized neighbor representation

    Inf. Sci.

    (2016)
  • J. Jiang et al.

    Face super-resolution via multilayer locality-constrained iterative neighbor embedding and intermediate dictionary learning

    IEEE Trans. Image Process.

    (2014)
  • J. Jiang et al.

    Noise robust face hallucination via locality-constrained representation

    IEEE Trans. Multimedia

    (2014)
  • J. Jiang et al.

    Facial image hallucination through coupled-layer neighbor embedding

    IEEE Trans. Circuits Syst. Video Technol.

    (2016)
  • Cited by (6)

    • OPLS-SR: A novel face super-resolution learning method using orthonormalized coherent features

      2021, Information Sciences
      Citation Excerpt :

      Many experimental results on different public databases have shown their stronger super-resolution capability, in contrast with reconstruction-based techniques. Up to now, researchers have developed numerous impressive learning-based image SR techniques [9–16]. For example, Freeman et al. [9] first proposed a learning-based image SR method in 2000, in which the mapping relationship from LR images to HR ones is learned by a patch-wise Markov network trained via a belief propagation algorithm.

    • Multi-view face hallucination using SVD and a mapping model

      2019, Information Sciences
      Citation Excerpt :

      Face hallucination is a technique to construct a high-resolution (HR) image from a low-resolution (LR) face image [30,33,40], and has become an active issue in computer vision and pattern recognition [9,10,21,24 46,48].

    • Face Super-resolution Based on Multi-source References

      2022, International Conference on Human System Interaction, HSI
    View full text