Morphable model space based face super-resolution reconstruction and recognition

Abstract

Super-resolution image reconstruction is the process of producing a high-resolution image from a set of low-resolution images of the same scene. For the applications of performing face evaluation and/or recognition from low-resolution video surveillance, in the past, super-resolution image reconstruction was mainly used as a separate preprocessing step to obtain a high-resolution image in the pixel domain that is later passed to a face feature extraction and recognition algorithm. Such three-stage approach suffers a high degree of computational complexity. A low-dimensional morphable model space based face super-resolution reconstruction and recognition algorithm is proposed in this paper. The approach tries to construct the high-resolution information both required by reconstruction and recognition directly in the low dimensional feature space. We show that comparing with generic pixel domain algorithms, the proposed approach is more robust and more computationally efficient.

Introduction

In many real applications, like performing face evaluation and/or recognition from low-resolution (LR) surveillance videos, we need to apply magnification to the LR face images. Nevertheless, generic magnification techniques usually result in blurred images, and unacceptable recognition rates can be expected if such blurred images are passed to a face recognition system. Therefore, super-resolution (SR) techniques have been proposed to solve this problem [1], [2], [3], [4].

SR image reconstruction is the process of recovering a high-resolution (HR) image from a single or a set of LR images. Single frame SR image reconstruction resolves high frequency information via various methods, e.g., probability density distribution of the scene [28], learning from examples [1]. The essential difference between single frame and multi-frame SR image reconstruction is that new high frequency information can also be recovered from different frames [22], [23]. For ideal cases, at most q² non-redundant LR frames (each frame has different motion vector, thus containing different sub-pixel information) are possible for resolution enhancement of q in each dimension. However, for real image sequences, it is usually almost impossible to acquire q² non-redundant frames except for the cases like translation-controlled imagers or similar systems.

Many SR algorithms have been proposed so far, including the frequency domain method, the projection onto convex sets (POCS) method, and the maximum a posteriori (MAP) method [5], [6], [7], [8], [9], [10], [11]. Among all the proposed SR algorithms, MAP has received much attention in recent years because it has many outstanding advantages, such as complete theory framework, flexible spatial domain observation model, powerful inclusion of a priori knowledge, and producing superior results.

To improve the quality of the reconstructed human face, Baker and Kanade [3] proposed to add a LR-HR face mapping based prior into the MAP framework. Capel and Zisserman[2] presented a similar approach, but the mapping was separately carried out for different regions of a face. Though all these methods [2], [3] can improve the quality of the final HR face image to some extend, the computational complexity and the sensitivity to noise are increased. In [12], [13], face SR reconstruction and recognition was performed directly in Eigenface space. Since Eigenface space is only a subspace generated by applying PCA analysis to the pixel space, the capability of representing a human face is worse than that of the pixel space. It can be expected that such methods should produce worse results than pixel-space-based methods. By learning the mapping among faces with different resolutions in a unified feature space, Li et al. [14] found that it was possible to carry out LR face recognition without any SR reconstruction preprocessing. Hennings–Yeomans et al. [15] proposed an approach for the recognition of LR faces by including the SR model and face features in a regularization framework. However, those methods [14], [15] are meant for recognition, not reconstruction.

Morphable model space is constructed by calculating dense pixelwise correspondence of different example images of objects of the same class, so it really contains the characteristics of shape and texture of the objects of a class. It was shown that with morphable model space, it is possible to construct a complete human face even when one quarter of the face information is lost [16], [17]. On the other hand, for the recognition of faces covering large variations in pose and illumination, V.Blanz and T.Vetter [18] got recognition rates above 95% in the morphable model space. All these researches [16], [17], [18] indicate that morphable model space is a powerful and versatile representation for human faces. Considering SR image reconstruction is also an information recovering process, through which low-frequency informations are used to resolve the hidden high-frequency informations, it is reasonable to expect better results when using the morphable model space to perform face SR reconstruction and recognition.

SR image reconstruction is also computationally expensive. Many works have been published to address this problem and can be roughly grouped into two categories. The first category was originated by Elad and Hel-or [24], which is characterized by braking the SR reconstruction into two consecutive stages: fusing and de-blurring, and applying fast algorithm to either or both of the two steps. Similar approaches include: S.Farsiu et al. [29] presented a fast implementation for the second step using L1 norm minimization; Miravet and Rodriguez [25] accelerated the process by using a hybrid neural network and a filtering operation for the first and the second step, respectively; M. Protter et al. [26] also suggested a fast nonlocal-means filtering algorithm for the first step. The second category speeds up the SR process by numeric techniques. For example, Hardie et al. [27] suggested the conjugate gradient (CG) method, Nguyen et al. [22] presented the preconditioned conjugate gradient (PCG) method.

Unlike the previous works, in this paper, we present a novel multi-frame face SR observation model between LR and HR morphable model space, and propose a complete MAP framework to solve this model. The proposed method also reduces the computational complexity by performing face reconstruction and recognition in a low-dimensional space.