Elsevier

Pattern Recognition

Volume 45, Issue 7, July 2012, Pages 2445-2459
Pattern Recognition

A comparative study of thermal face recognition methods in unconstrained environments

https://doi.org/10.1016/j.patcog.2012.01.001Get rights and content

Abstract

The recognition of faces in unconstrained environments is a challenging problem. The aim of this work is to carry out a comparative study of face recognition methods working in the thermal spectrum (8–12 μm) that are suitable for working properly in these environments. The analyzed methods were selected by considering their performance in former comparative studies, in addition to being real-time, to requiring just one image per person, and to being fully online (no requirements of offline enrollment). Thus, in this study three local-matching methods based on histograms of Local Binary Pattern (LBP) features, on histograms of Weber Linear Descriptors (WLD), and on Gabor Jet Descriptors (GJD), as well as two global image-matching method based on Scale-Invariant Feature Transform (SIFT) Descriptors, and Speeded Up Robust Features (SURF) Descriptors, are analyzed. The methods are compared using the Equinox and UCHThermalFace databases. The use of these databases allows evaluating the methods in real-world conditions that include natural variations in illumination, indoor/outdoor setup, facial expression, pose, accessories, occlusions, and background. The UCHThermalFace database is described for the first time in this article and WLD is used for the first time in face recognition. The results of this comparative study are intended to be a guide for developers of face recognition systems. The main conclusions of this study are: (i) all analyzed methods perform very well under the conditions in which they were evaluated, except for the case of GJD that has low performance in outdoor setups; (ii) the best tradeoff between high recognition rate and fast processing speed is obtained by WLD-based methods, although the highest recognition rate in all cases is obtained by SIFT-based methods; and (iii) in experiments where the test images are acquired in an outdoor setup and the gallery images are acquired in an indoor setup, or vice versa, the performance of all evaluated methods is very low. As part of the future work, the use of normalization algorithms and calibration procedures in order to tackle this last issue will be analyzed.

Highlights

► A comparison of thermal face-recognition methods is presented. ► The UCHThermalFace database is described for the first time. ► WLD is used for the first time in face recognition. ► All analyzed methods perform very well under most of the evaluation. ► The best tradeoff between recognition rate and processing speed is obtained by WLD.

Introduction

The recognition of human faces in unconstrained environments has attracted increasing interest in the research community in recent years. Several studies have shown that the use of thermal images can solve limitations of visible-spectrum based face recognition, such as invariance to variations in illumination and robustness to variations in pose [37], [38], which are two of the major factors affecting the performance of face recognition systems in unconstrained environments [36]. This is possible, thanks to the physical properties of thermal technology (long-wave infrared spectrum, 8–12 μm), and the anatomic characteristics of the human body:

  • Thermal sensors collect the energy emitted by a body instead of the reflected light, and the emissivity of human skin is between 8–12 μm,

  • thermal sensors are invariant to changes in illumination; they can even work in complete darkness, and

  • the anatomic and vascular information that can be extracted from thermal images is unique to each individual [13].

In addition, in recent years, the price of thermal cameras has decreased significantly, and their technology has improved, obtaining better resolution and quality, and the fixed pattern noise that was produced by old thermal cameras has been eliminated using non-uniformity correction techniques (NUC) [28], [29]. Thus, the interest in the use of thermal technology in face recognition applications has increased in recent years. Nevertheless, thermal face images still have undesirable variations due to (i) changes in ambient temperature, (ii) modifications of the metabolic processes of the subjects, (iii) camera susceptibility on extrinsic factors such as wind, and (iv) variable sensor response overtime when the camera is working for long periods of times [10], [13], [39].

In this general context, the aim of this article is to carry out a comparative study of thermal face-recognition methods in unconstrained environments. The main motivation is the lack of direct1 and detailed comparisons of these kinds of method working under the same conditions. The results of this comparative study are intended to be a guide for developers of face recognition systems. This study concentrates on methods that fulfill the following requirements: (i) Full online operation: No offline enrollment stages. All processes must run online. The system has to be able to build the face database incrementally from scratch; (ii) Real-time operation: The recognition process should be fast enough to allow real-time interaction in case of HRI (Human-Robot Interaction) applications, and to search large databases in a reasonable time (a few milliseconds or some seconds depending on the application and the size of the database); (iii) Single image per person problem: One thermal face image of an individual should be enough for his/her later identification. Databases containing just one face image per person should be considered. The main reasons for this are savings in storage and computational costs, and the impossibility of obtaining more than one face image from a given individual in certain situations; and (iv) Unconstrained environments: No restrictions on environmental conditions such as illumination, indoor/outdoor setup, facial expression, scale, pose, resolution, accessories, occlusions, and background are imposed.

Thus, in this study three local-matching and two global image-matching method are selected by considering their fulfillment of the previously mentioned requirements, and their good performance in former comparative studies of face recognition methods [35], [36], [41], [46]. Two local-matching methods, namely, histograms of LBP (Local Binary Pattern) features [3] and Gabor Jet Descriptors with Borda count classifiers [46] are selected based on their performance in the studies reported in [36], [46]. The third local-matching method, histograms of WLD (Weber Linear Descriptor) features, recently proposed in [12], has shown very good performance in face detection applications, and is used here for the first time in face recognition. The SIFT (Scale-Invariant Feature Transform) image-matching method [26] is included following its good performance in a former face recognition study [36]. Finally, the SURF (Speeded Up Robust Features) image-matching method [5], which is inspired by the SIFT method, is included because in many applications in which the SIFT method is used, SURF obtains a similar performance and a higher speed.

The comparative study is carried out using the Equinox and UCHThermalFace databases. The Equinox database [16] was selected because it is one of the most frequently employed thermal face databases, and therefore it allows comparing the obtained results with former studies. The UCHThermalFace database was specially designed to study the problem of unconstrained face recognition in the thermal domain. The database incorporates thermal images acquired in indoor and outdoor setups, with natural variations in illumination, facial expression, pose, accessories, occlusions, and background. This database will be made public for future comparative studies, which is also a contribution of this paper.

This comparative study intends to be a complement to the recently published comparative study on visible-spectrum face recognition methods in unconstrained environments [36].

The paper is structured as follows: Related works are outlined in Section 2. The methods under analysis are described in Section 3. In 4 Comparative study using the equinox database, 5 Comparative study using the UCHThermalFace database the comparative analyses of these methods in the Equinox and UCHThermalFace databases are presented. Finally, in Section 6 results are discussed, and conclusions are given.

Section snippets

Related work

Several comparative studies of thermal face recognition approaches have been developed in recent years [37], [38], [40]. Most of the developed approaches make use of appearance-based methods, such as PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), and ICA (Independent Component Analysis), which project face images into a subspace where the recognition is carried out. These methods achieve a ∼95% recognition rate in experiments that do not consider real-world conditions

Methods under comparison

As mentioned above, the methods under comparison were selected considering their fulfillment of the defined requirements (real-time, fully online, just one image per person), and their performance in former comparative studies of face-recognition methods [35], [36], [41], [46] and in face detection applications [12].

Comparative study using the equinox database

The performance of the methods is analyzed using the Equinox database [16], which has already been used to validate several face recognition methods for thermal images. This allows the direct comparison of the selected methods with face recognition methods implemented in previous works: in [37], [38] methods based on LDA (Linear Discriminant Analysis), LFA (Local Feature Analysis), and ICA (Independent Component Analysis) have been compared. PCA (Principal Component Analysis) has been analyzed

Comparative study using the UCHThermalFace database

The methods under study are analyzed considering real-world conditions that include indoor/outdoor setups and natural variations on facial expression, pose, accessories, occlusions, and background.

Discussion and conclusions

In this article, a comparative study of thermal-based face-recognition methods in unconstrained environments was presented. The analyzed methods were selected by considering their suitability for the defined requirements —real-time operation, just one image per person, fully online (no training), and robust behavior in unconstrained environments—and their performance in former studies. The comparative study was carried out using two databases: Equinox and UCHThermalFace. The well-known Equinox

Acknowledgements

This research was partially funded by the FONDECYT-Chile Grant 1090250 and by the Advanced Mining Technology Center.

References (44)

  • P. Buddharaju et al.

    Multi-spectral face recognition—fusion of visual imagery with physiological information

  • P. Buddharaju et al.

    Face recognition beyond the visible spectrum, advances in biometrics: sensors

    Algorithms and Systems

    (2007)
  • P. Buddharaju, I. Pavlidis, I. Kakadiaris, Physiology-based face recognition, in: Proceedings of the IEEE Conference on...
  • P. Buddharaju, I. Pavlidis, I. Kakadiaris, Pose-invariant physiological face recognition in the thermal infrared...
  • J. Chen et al.

    WLD: a robust local image descriptor

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2010)
  • X. Chen, P. Flynn, K.W. Bowyer, PCA-based face recognition in infrared imagery: baseline and comparative studies, IEEE...
  • S.Y. Cho, L. Wang, W.L. Ong, Thermal imprint feature analysis for face recognition, Industrial Electronics, 2009. ISIE...
  • S. Desa, S. Hati, IR and visible face recognition using fusion of kernel based features, In: Proceedings of the 19th...
  • Equinox, 2011. Equinox Database. Available on July 2011 in:〈...
  • R. Gross

    Face databases

  • G. Hermosilla, P. Loncomilla, J. Ruiz-del-Solar, Thermal Face Recognition using Local Interest Points and Descriptors...
  • G. Hermosilla, J. Ruiz-del-Solar, R. Verschae, M. Correa, Face recognition using thermal infrared images for...
  • Cited by (89)

    • Attacks and defenses in user authentication systems: A survey

      2021, Journal of Network and Computer Applications
    • An extensive review on spectral imaging in biometric systems: Challenges & advancements

      2019, Journal of Visual Communication and Image Representation
    View all citing articles on Scopus
    View full text