A Two-Phase Weighted Collaborative Representation for 3D partial face recognition with single sample

Highlights

• Novel Keypoint-based Multiple Triangle Statistics (KMTS) are proposed for 3D face representation.

• The proposed local descriptor is robust to partial facial data and expression/pose variations.

• A Two-Phase Weighted Collaborative Representation Classification (TPWCRC) framework is used to perform face recognition.

• The proposed classification framework can effectively address the single sample problem.

• State-of-the-art performance on six challenging datasets with high efficiency is achieved.

Abstract

3D face recognition with the availability of only partial data (missing parts, occlusions and data corruptions) and single training sample is a highly challenging task. This paper presents an efficient 3D face recognition approach to address this challenge. We represent a facial scan with a set of local Keypoint-based Multiple Triangle Statistics (KMTS), which is robust to partial facial data, large facial expressions and pose variations. To address the single sample problem, we then propose a Two-Phase Weighted Collaborative Representation Classification (TPWCRC) framework. A class-based probability estimation is first calculated based on the extracted local descriptors as a prior knowledge. The resulting class-based probability estimation is then incorporated into the proposed classification framework as a locality constraint to further enhance its discriminating power. Experimental results on six challenging 3D facial datasets show that the proposed KMTS–TPWCRC framework achieves promising results for human face recognition with missing parts, occlusions, data corruptions, expressions and pose variations.

Introduction

Face recognition (FR) is an active research area in the computer vision community due to its non-intrusive and friendly acquisition nature when compared to other biometrics. The task of FR is to identify or verify a human face from its records (2D and 3D modalities). FR has a number of real-world applications including access control and video surveillance. Among various facial modalities, considerable progress has been made with 2D FR. However, its performance is still challenged by pose changes and illumination variations. With the rapid development of 3D acquisition technologies [1], [2], [3], [4], [5], [6], [7], 3D FR has drawn growing attention due to its potential capability to overcome the inherent limitations of its 2D counterpart. Most of the existing 3D FR approaches are proposed for facial scans acquired in a highly controlled environments. Such facial scans are mostly frontal and good in quality. Promising recognition performance has been reported on those high quality 3D facial data by existing 3D FR approaches. However, for many real-world applications, facial scans can only be acquired in uncontrolled environments. Such facial scans usually contain a partial face in the presence of missing parts, occlusions and data corruptions (as shown in Fig. 1). FR from such partial facial scans is still an open issue, and further investigation is required to enable fully automatic 3D Partial Face Recognition (PFR). Although a few 3D PFR approaches have been proposed, the recent release of datasets containing partial facial scans (e.g., the Bosphorus [8], GavabDB [9] and UMB-DB [10] datasets) provides a large benchmark for 3D PFR and has further boosted the research on this topic. Furthermore, most of the existing 2D/3D FR approaches require a sufficiently large set of training samples per individual to cover possible facial variations (e.g., partial data and expressions) for accurate FR. However, in many real-world FR applications, only a single sample per individual can be provided for training, resulting in the single sample based FR problem. Partial facial data and the limitation in the number of training samples per individual are therefore two important challenges for real-world 3D PFR applications.

The existing 3D FR approaches can be coarsely classified into two categories: global descriptor based and local descriptor based approaches. The global descriptor based approaches extract features from the entire face to encode the geometric characteristics of a 3D face [5], [7]. However, these approaches only work when the complete 3D face is available. These approaches therefore rely on the availability of complete facial scan and are sensitive to missing parts, occlusions and data corruptions. In contrast, local descriptor based approaches usually extract features from the neighborhoods of the detected keypoints, resulting in a set of coordinate-independent local descriptors. Compared to global descriptor based approaches, the number of extracted local descriptors is related to the content of input face (holistic or partial). Therefore, such flexible facial representation is more suitable for 3D PFR. Furthermore, it is commonly believed that only a few facial regions are significantly affected by distortions caused by missing parts, occlusions and data corruptions, while most of the other regions remain invariant [11], [12]. This indicates that local facial descriptors are more suitable and powerful when dealing with partial facial data.

Recently, Sparse Representation (SR) has become a powerful method for various pattern recognition tasks. Experimental results on FR show that the Sparse Representation-based Classification (SRC) algorithms outperform a number of conventional FR algorithms. However, solving l₁-norm based sparsity usually requires computationally demanding optimization procedures. It is argued that the l₁-norm sparsity is not essential for the improvement of FR performance [13]. Several more efficient algorithms have therefore been proposed, including the Collaborative Representation-based Classification (CRC), which can achieve similar recognition performance compared to SRC [13], [14]. However, both SRC and CRC require multiple training samples for each individual to cope with the test input with complex variations. In the case of single sample problem, the representation becomes unreliable due to the lack of training samples. Inspired by this fact, more prior classification knowledge can be incorporated into the data representation to perform single sample based FR.

In this paper, we propose a Two-Phase Weighted Collaborative Representation Classification (TPWCRC) for single sample based 3D PFR. First, an efficient Keypoint-based Multiple Triangle Statistics (KMTS) descriptor is presented for facial scan representation (Section 3). The proposed KMTS descriptor exhibits both high discriminative power and robustness under various deformations (i.e., partial data, expressions and pose variations). Second, a class-based probability estimation algorithm is presented to provide a strong prior classification knowledge for any input test during the first phase of classification. The probability estimation can then be used to compensate for the unavailability of multiple training samples per individual. The resulting probability estimation is therefore integrated into the TPWCRC framework as a locality constraint to restrain the globality of the data representation of the input test. The final classification result then corresponds to the individual which gives the smallest residuals (Section 4). The proposed KMTS–TPWCRC 3D PFR approach is alignment-free and no manual intervention is required. Its efficiency and robustness has been extensively demonstrated by a set of experiments on six popular 3D face datasets (Section 5).

Furthermore, the proposed approach only involves simple computations, including nosetip detection, keypoint detection, local geometrical descriptor and a TPWCRC framework which can be derived analytically. For the considerations of developing an efficient 3D PFR system, simple geometric features are extracted without any complex mathematical operations. Unlike many existing 3D PFR approaches which rely on complicated and time-consuming feature extraction methods (e.g., [15], [16]) or occlusion data restoration method (e.g., [15], [17]), the proposed approach is very efficient (see Section 5.5). Its efficiency can further be improved by performing these simple computations in parallel. Although some existing methods achieve better FR performance, the proposed approach can achieve a comparable performance at much lower computational cost. This suggests that our proposed approach is more suitable for practical applications.

The rest of this paper is organized as follows. Section 2 provides a brief literature review of closely related 3D FR approaches. Section 3 describes the proposed KMTS facial descriptors. Section 4 introduces the proposed TPWCRC framework for single sample based 3D PFR. Section 5 presents the experimental results and comparative analysis on five popular 3D facial datasets. Section 6 concludes the paper.