A new approach for face recognition by sketches in photos

doi:10.1016/j.sigpro.2009.02.008

Signal Processing

Volume 89, Issue 8, August 2009, Pages 1576-1588

https://doi.org/10.1016/j.sigpro.2009.02.008 Get rights and content

Abstract

Face recognition by sketches in photos remains a challenging task. Unlike the existing sketch–photo recognition methods, which convert a photo into sketch and then perform the sketch–photo recognition through sketch–sketch recognition, this paper devotes to synthesizing a photo from the sketch and transforming the sketch–photo recognition to photo–photo recognition to achieve better performance in mixture pattern recognition. The contribution of this paper mainly focuses on two aspects: (1) in view of that there are no many research findings of sketch–photo recognition based on the pseudo-photo synthesis and the existing methods require a large set of training samples, which is nearly impossible to achieve for the high cost of sketch acquisition, we make use of embedded hidden Markov model (EHMM), which can learn the nonlinearity of sketch–photo pair with less training samples, to produce pseudo-photos in terms of sketches; and (2) photos and sketches are divided into patches and pseudo-photo is generated by combining pseudo-photo patches, which makes pseudo-photo more recognizable. Experimental results demonstrate that the newly proposed method is effective to identify face sketches in photo set.

Introduction

As one of the personal identification methods with the most potential, face recognition [1], [2], [3] has drawn increasing attention of researchers and played an important role in many application areas, one of which is retrieving a suspect's images from the existing photo database automatically for identifying the suspect so as to prevent terrorist activities and ensure public security. An automated retrieval of suspects’ images would not only save many person-hours, but also reduce the effect of a subjective assessment. Actually, we acquire no photos of suspects but witnesses’ verbal description of them sometimes; thereby simulated sketches are generated by the cooperation of artists and witnesses instead of photos, and identity of suspects has to be determined by searching the existing photo database based on sketches. The problem of recognizing sketches from photo database automatically and effectively has attracted researchers. Since the research of face recognition sprang up in 1960s, many significant achievements have been made [1], [2], [3] and most recently [4], [5], [6], [7], [8], [9], and most of them are only available for face photos. Sketches and photos are generated and expressed in different mechanism; therefore have great geometrical deformations and large difference of texture. In other words, the sketch and photo for a person may be similar in geometry, but the texture of them is always very different, which makes the existing face recognition algorithms inactive. Sketch–photo recognition becomes a challenging research focus of face recognition and deserves further research.

The key of sketch–photo recognition method is transforming photos and sketches into the same modality to reduce difference between them, and then face recognition by sketches in pseudo-sketches or by pseudo-photos in photos is performed using classical approaches [10], [11]. The initial research is presented in [12], [13], which rely on human intervention heavily. Robert et al. proposed the method for automatic match of sketches and photos firstly [14]. As shown in Fig. 1, up to now, face sketches referred to in sketch–photo recognition focus on line-drawing sketch and complex sketch which depicts the face information by lighting and shading besides lines. In line-drawing sketch–photo recognition, photos have to be converted into line-drawing sketches, with which the line-drawing sketch to be recognized is compared. For complex sketch–photo recognition, on the one hand, photos are converted into pseudo-sketches based on which identification of complex sketch is performed. Tang et al. have made large contribution to convert photos into pseudo-sketches. They proposed a method based on principal components analysis (PCA) [15], [16], [17], and then introduced manifold in the pseudo-sketch synthesis [18] in which nonlinearity between photos and sketches is approximated by local linearity. Extending these ideas, Gao et al. proposed sketch synthesis algorithms [19], [20], [21] based on embedded hidden Markov model (EHMM) so that complex nonlinear relationship between photos and sketches is learnt exactly. Pseudo-sketch synthesis is developed from linear method, nonlinear method approximated by local linearity to real nonlinear method. On the other hand, in view of much information useful for recognition may be lost if all face photos are transformed into sketches, the complex sketch is transformed into a pseudo-photo which is recognized in photo databases. Facial pseudo-photo synthesis are achieved by photometrically standardizing sketch images [14], [22], using a hybrid subspace method [23], and based on Bayesian tensor inference [24]. The trend in development of these methods is from the method based on basic pixels to the statistical one through the method with subspace. Our aim is the recognition of complex sketch in photo database and in the following, complex sketch is abbreviated to sketch.

The research of sketch–photo recognition based on synthesizing pseudo-photos from the corresponding sketches is still at an elementary stage. The existing methods require a large set of training samples; however, the training set is usually restricted within a small size because of the high cost of sketch acquisition. The application of this kind of methods is limited. Aiming at this problem, a novel face pseudo-photo synthesis algorithm is proposed based on machine learning and recognition of pseudo-photo in photos is performed in this paper so as to implement sketch–photo recognition with small set of training samples. The nonlinear relationship of each sketch–photo pair in training set is learnt by a pair of EHMMs [25]. Given a face sketch, several intermediate pseudo-photos for it are generated based on the selected trained EHMM pairs according to the idea of selective ensemble and the expected pseudo-photo is resulted by fusing these intermediate pseudo-photos. Each intermediate pseudo-photo is generated with a training sketch–photo pair and it is a pseudo-photo for the given sketch to a certain extent, that is to say, one pair of training sketch and photo is sufficient for obtaining a pseudo-photo. Several training sketch–photo pairs are chosen to synthesize more intermediate pseudo-photos for the given sketch, so that they can be fused to generate a much better pseudo-photo. Accordingly, it is unnecessary to collect a great many of training samples. In addition, compared with the whole face, local facial features can provide more specific information, which are in favor of state estimation of EHMMs, and then we add local strategy to the synthesis of pseudo-photo. When the pseudo-photo is synthesized, eigenface recognition [10], [26] is performed between pseudo-photo and photos. It can be proved that the proposed method leads to high recognition rate of sketch–photo recognition with a set of experiments.

The remainder of this paper is organized as follows. Section 2 gives an overview to the proposed algorithm. In Section 3, the idea of EHMM is introduced briefly, and an EHMM pair is modeled for a pair of sketch patch and photo patch. The procedure of synthesizing and identifying pseudo-photos is described in detail in Section 4. The experimental results are presented in Section 5, and the final section gives the conclusions.

Section snippets

The overview of the proposed method

The aim of the proposed method is to identify a sketch in a photo set by synthesizing a pseudo-photo for the sketch and performing photo–photo recognition. Hidden Markov model (HMM) is a probability statistics method used to model signals for processing [27]. Because face images have rich two-dimensional spatial information, modeling a face image with the traditional one-dimensional HMM not only loses the spatial information partially, but also increases computational complexity which is still

Embedded HMMs pair of sketch patch and photo patch

EHMM consists of a series of super-states, each of which contains several embedded-states. Similar to the HMM, states of the EHMM are non-observable and only a sequence of instances $O = {o_{1}, o_{2}, \dots, o_{T}}$ generated by these states can be observed, T is the number of observation vectors. The model is denoted as $λ = (Π_{s}, A_{s}, Λ_{e}, N_{s}),$ where $Π_{s} = {Π_{k}}$ includes initial super-state distribution vectors and $Π_{k}$ is for the k-th super-state, $A_{s} = {a_{kq}}$ is super-state transition probability matrix representing the

The synthesis and recognition of pseudo-photos

The flow chart of synthesizing pseudo-photos and performing recognition of pseudo-photos in photos is shown in Fig. 4. A training set with M pairs of sketches and photos $(S_{i}, P_{i})$ , where $i = 1, 2, \dots, M$ , and a sketch S to be recognized are given. Firstly, they are evenly divided into N overlapping patches with the patch size B×B and the overlapped area size $B \times B \times D$ , where D is the overlapping degree. Each patch $s_{trj}$ of sketches in the training set is modeled with EHMM $λ_{trj}$ , where $j = 1, 2, \dots, M \times N$ ; Secondly,

The experimental results and analysis

In this section, the effectiveness of the proposed sketch–photo recognition algorithm is evaluated from the aspect of identifying sketch in photo set. The proposed method is compared with the direct sketch–photo recognition method and the approach based on pseudo-sketch synthesis [21]. In the direct sketch–photo recognition method, the testing sketch is identified based on training photos with eigenface method. According to the approach reported in [21] whose idea is illustrated in Fig. 5,

Conclusion

This paper presents a novel sketch–photo recognition method. The sketch to be recognized is converted into a pseudo-photo by the combination of EHMM and selective ensemble with the local strategy. The nonlinear relationship between sketch and photo patch pairs is modeled using EHMMs, many of which are selected to reconstruct pseudo-photo patches. These pseudo-photo patches are synthesized into the expected pseudo-photo. Then sketch–photo recognition is transformed into recognizing pseudo-photo

Acknowledgements

The authors are grateful to the helpful comments and suggestions from the anonymous reviewers. Thanks must be expressed to the Multimedia Lab of the Chinese University of Hong Kong for providing us with the face photo–sketch images database. This research was supported by National Science Foundation of China (60771068, 60702061, 60832005), the Open-End Fund of National Laboratory of Pattern Recognition in China and National Laboratory of Automatic Target Recognition, Shenzhen University, China,

References (41)

X. Gao et al.
Local face sketch synthesis learning
Neurocomputing
(2008)
X. Li et al.
Adaptive color quantization based on perceptive edge protection
Pattern Recognition Lett.
(2003)
R. Chellappa et al.
Human and machine recognition of faces: a survey
Proc. IEEE
(1995)
W. Zhao et al.
Face recognition: a literature survey
ACM Comput. Surv.
(2003)
P. Phillips, P. Flynn, T. Scruggs, K. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, W. Worek, Overview of the face...
Y. Pang et al.
Binary two-dimensional PCA
IEEE Trans. Syst. Man Cybern. B Cybern.
(2008)
D. Tao et al.
Bayesian tensor approach for 3-D face modeling
IEEE Trans. Circuits Syst. Video Technol.
(2008)
Y. Pang et al.
Gabor-based region covariance matrices for face recognition
IEEE Trans. Circuits Syst. Video Technol.
(2008)
D. Tao et al.
General tensor discriminant analysis and Gabor features for gait recognition
IEEE Trans. Pattern Anal. Mach. Intell.
(2007)
D. Tao et al.
Geometric mean for subspace selection
IEEE Trans. Pattern Anal. Mach. Intell.
(2009)

T. Zhang, D. Tao, J. Yang, Discriminative locality alignment, in: Proceedings of European Conference on Computer...

M. Turk, A. Pentland, Face recognition using eigenfaces, in: Proceedings of IEEE Conference on Computer Vision and...

M. Bartlett et al.

Face recognition by independent component analysis

IEEE Trans. Neural Networks

(2002)

A. Narasimhalu, CAFFIR: an image based CBR/IR application, in: Proceedings of AAA1 Spring Symposium,...

J. Shepherd

An interactive computer system for retrieving faces

Robert G. Uhl Jr., Niels da Vitoria Lobo, A framework for recognizing a facial image from a police sketch, in:...

X. Tang, X. Wang, Face photo recognition using sketch, in: Proceedings of IEEE International Conference on Image...

X. Tang, X. Wang, Face sketch synthesis and recognition, in: Proceedings of IEEE International Conference on Computer...

X. Tang et al.

Face sketch recognition

IEEE Trans. Circuits Syst. Video Technol.

(2004)

Q. Liu, X. Tang, H. Jin, H. Lu, S. Ma, A nonlinear approach for face sketch synthesis and recognition, in: Proceedings...

Cited by (81)

Face image-sketch synthesis via generative adversarial fusion
2022, Neural Networks
Citation Excerpt :
Moreover, these existing methods ignore learning and controlling the illumination distribution over the face. It is easy to observe that the face area of generated face images has no illumination highlight (Duan et al., 2021; Wang et al., 2018; Xiao et al., 2009) or has stronger illumination highlight (Sannidhan et al., 2019; Wang & Tang, 2009; Zhu et al., 2019) compared with the ground truth image. The illumination highlight is the key factor that directly affects the visual quality of the generated face image.
Face image-sketch synthesis is widely applied in law enforcement and digital entertainment fields. Despite the extensive progression in face image-sketch synthesis, there are few methods focusing on generating a color face image from a sketch. The existing methods pay less attention to learning the illumination or highlight distribution on the face region. However, the illumination is the key factor that makes the generated color face image looks vivid and realistic. Moreover, existing methods tend to employ some image preprocessing technologies and facial region patching approaches to generate high-quality face images, which results in the high complexity and memory consumption in practice. In this paper, we propose a novel end-to-end generative adversarial fusion model, called GAF, which fuses two U-Net generators and a discriminator by jointly learning the content and adversarial loss functions. In particular, we propose a parametric tanh activation function to learn and control illumination highlight distribution over faces, which is integrated between the two U-Net generators by an illumination distribution layer. Additionally, we fuse the attention mechanism into the second U-Net generator of GAF to keep the identity consistency and refine the generated facial details. The qualitative and quantitative experiments on the public benchmark datasets show that the proposed GAF has better performance than existing image-sketch synthesis methods in synthesized face image quality (FSIM) and face recognition accuracy (NLDA). Meanwhile, the good generalization ability of GAF has also been verified. To further demonstrate the reliability and authenticity of face images generated using GAF, we use the generated face image to attack the well-known face recognition system. The result shows that the face images generated by GAF can maintain identity consistency and well maintain everyone’s unique facial characteristics, which can be further used in the benchmark of facial spoofing. Moreover, the experiments are implemented to verify the effectiveness and rationality of the proposed parametric tanh activation function and attention mechanism in GAF.
Local Force Pattern (LFP): Descriptor for Heterogeneous Face Recognition
2019, Pattern Recognition Letters
Citation Excerpt :
Similarly, Liu et al. [11] used Locally Linear Embedding (LLE) to synthesise sketches. Xiao et al. [12] stated an approach based on Embedded Hidden Markov Model (E-HMM) for nonlinear transformation of sketches into pseudo-photo, however this approach is computationally expensive because it is an iterative method. In the field of feature representation, Yuen et al. [13] proposed an approach for representing local facial feature and geometrical relationship between facial features as a global representation of face sketch and photos.
This paper proposes a novel local-appearance based feature descriptor namely Local Force Pattern (LFP), for robust heterogeneous face recognition. LFP features encode the directional information of the textures of the given face images compactly, producing a more discriminating code than the state of the art methods. We compute the structure of each micro-pattern with the aid of Van der Waal’s force between a pair of pixels, which extracts the directional information, and encodes it using a 2-D vector. This directional information helps us to discriminate images having similar structural patterns with different modality based intensity transitions. We divide the face image into several small pixel boxes called ‘cell’ and extract the LFP features from them. Several cell forms ‘block’. The resultant force obtained from the block is vector-summed to get the final feature vector. Then, we concatenate these features into a vector and use it as the face descriptor. Experiments show that the proposed descriptor consistently performs well over state-of-the-art methods for different heterogeneous face datasets like CUSF, IIIT-D and CASIA.
Feature fusion with covariance matrix regularization in face recognition
2018, Signal Processing
The fusion of multiple features is important for achieving state-of-the-art face recognition results. This has been proven in both traditional and deep learning approaches. Existing feature fusion methods either reduce the dimensionality of each feature first and then concatenate all low-dimensional feature vectors, named as DR-Cat, or the vice versa, named as Cat-DR. However, DR-Cat ignores the correlation information between different features which is useful for classification. In Cat-DR, on the other hand, the correlation information estimated from the training data may not be reliable especially when the number of training samples is limited. We propose a covariance matrix regularization (CMR) technique to solve problems of DR-Cat and Cat-DR. It works by assigning weights to cross-feature covariances in the covariance matrix of training data. Thus the feature correlation estimated from training data is regularized before being used to train the feature fusion model. The proposed CMR is applied to 4 feature fusion schemes: fusion of pixel values from 3 color channels, fusion of LBP features from 3 color channels, fusion of pixel values and LBP features from a single color channel, and fusion of CNN features extracted by 2 deep models. Extensive experiments of face recognition and verification are conducted on databases including MultiPIE, Georgia Tech, AR and LFW. Results show that the proposed CMR technique significantly and consistently outperforms the best single feature, DR-Cat and Cat-DR.
Face sketch-photo synthesis and recognition: Dual-scale Markov Network and multi-information fusion
2018, Journal of Visual Communication and Image Representation
Sketch face recognition (SFR) has been widely and successfully applied in law enforcement, which attracts a growing number of researchers. In this paper, a face sketch-photo synthesis and recognition method is proposed. Our method has two parts: Firstly, according to the different synthesis results for different scales, a cascade sketch-photo synthesis method via dual-scale Markov Network is utilized for image synthesis; Secondly, structural information and feature information-based data fusion method has been presented for face recognition. It is inspired by the Face Recognition Cognitive Theory, which applies both structural information and feature information for recognition. The experimental results on different databases based on the proposed method, demonstrate the outperformance of our method compared with state-of-the-art methods both in synthesis and recognition processes.
Fast alignment for sparse representation based face recognition
2017, Pattern Recognition
Sparse representation based face recognition (SRC) has been paid much attention in recent years. Representative algorithms are deformable sparse-representation based classification (DSRC) and shape-constrained texture matching, which focus on misalignment and shape change respectively. Although these algorithms obtain improved accuracy and robustness, their efficiency is not satisfied particularly when applied in a large-scale system. The main difficulty is the expensive calculation in aligning gallery images and the probe image. To solve these problems, this paper proposes a fast alignment strategy for sparse representation based algorithms. The key idea is to pre-compute the most expensive operation, Hessian matrix, which was needed to be calculated in each iteration. Subsequently, with help of the proposed fast alignment strategy, two algorithms, deformable SRC and shape-constrained texture matching, are extended to their fast versions, i.e., fast deformable SRC and fast shape-constrained texture matching. Experimental evaluations have been conducted on the public datasets such as Multi-PIE, FERET, and Cohn-Kanade. We demonstrate that the proposed alignment strategy greatly improves the efficiency of original algorithms without losing accuracy and robustness.
A Deep Learning-Based Approach for Identification and Recognition of Criminals
2023, International Journal of Intelligent Systems and Applications in Engineering

View all citing articles on Scopus

View full text

A new approach for face recognition by sketches in photos

Abstract

Introduction

Section snippets

The overview of the proposed method

Embedded HMMs pair of sketch patch and photo patch

The synthesis and recognition of pseudo-photos

The experimental results and analysis

Conclusion

Acknowledgements

Neurocomputing

Pattern Recognition Lett.

Human and machine recognition of faces: a survey

Proc. IEEE

Face recognition: a literature survey

ACM Comput. Surv.

Binary two-dimensional PCA

IEEE Trans. Syst. Man Cybern. B Cybern.

Bayesian tensor approach for 3-D face modeling

IEEE Trans. Circuits Syst. Video Technol.

Gabor-based region covariance matrices for face recognition

IEEE Trans. Circuits Syst. Video Technol.

General tensor discriminant analysis and Gabor features for gait recognition

IEEE Trans. Pattern Anal. Mach. Intell.

Geometric mean for subspace selection

IEEE Trans. Pattern Anal. Mach. Intell.

Face recognition by independent component analysis

IEEE Trans. Neural Networks

An interactive computer system for retrieving faces

Face sketch recognition

IEEE Trans. Circuits Syst. Video Technol.