A comparative study of thermal face recognition methods in unconstrained environments
Highlights
► A comparison of thermal face-recognition methods is presented. ► The UCHThermalFace database is described for the first time. ► WLD is used for the first time in face recognition. ► All analyzed methods perform very well under most of the evaluation. ► The best tradeoff between recognition rate and processing speed is obtained by WLD.
Introduction
The recognition of human faces in unconstrained environments has attracted increasing interest in the research community in recent years. Several studies have shown that the use of thermal images can solve limitations of visible-spectrum based face recognition, such as invariance to variations in illumination and robustness to variations in pose [37], [38], which are two of the major factors affecting the performance of face recognition systems in unconstrained environments [36]. This is possible, thanks to the physical properties of thermal technology (long-wave infrared spectrum, 8–12 μm), and the anatomic characteristics of the human body:
- –
Thermal sensors collect the energy emitted by a body instead of the reflected light, and the emissivity of human skin is between 8–12 μm,
- –
thermal sensors are invariant to changes in illumination; they can even work in complete darkness, and
- –
the anatomic and vascular information that can be extracted from thermal images is unique to each individual [13].
In addition, in recent years, the price of thermal cameras has decreased significantly, and their technology has improved, obtaining better resolution and quality, and the fixed pattern noise that was produced by old thermal cameras has been eliminated using non-uniformity correction techniques (NUC) [28], [29]. Thus, the interest in the use of thermal technology in face recognition applications has increased in recent years. Nevertheless, thermal face images still have undesirable variations due to (i) changes in ambient temperature, (ii) modifications of the metabolic processes of the subjects, (iii) camera susceptibility on extrinsic factors such as wind, and (iv) variable sensor response overtime when the camera is working for long periods of times [10], [13], [39].
In this general context, the aim of this article is to carry out a comparative study of thermal face-recognition methods in unconstrained environments. The main motivation is the lack of direct1 and detailed comparisons of these kinds of method working under the same conditions. The results of this comparative study are intended to be a guide for developers of face recognition systems. This study concentrates on methods that fulfill the following requirements: (i) Full online operation: No offline enrollment stages. All processes must run online. The system has to be able to build the face database incrementally from scratch; (ii) Real-time operation: The recognition process should be fast enough to allow real-time interaction in case of HRI (Human-Robot Interaction) applications, and to search large databases in a reasonable time (a few milliseconds or some seconds depending on the application and the size of the database); (iii) Single image per person problem: One thermal face image of an individual should be enough for his/her later identification. Databases containing just one face image per person should be considered. The main reasons for this are savings in storage and computational costs, and the impossibility of obtaining more than one face image from a given individual in certain situations; and (iv) Unconstrained environments: No restrictions on environmental conditions such as illumination, indoor/outdoor setup, facial expression, scale, pose, resolution, accessories, occlusions, and background are imposed.
Thus, in this study three local-matching and two global image-matching method are selected by considering their fulfillment of the previously mentioned requirements, and their good performance in former comparative studies of face recognition methods [35], [36], [41], [46]. Two local-matching methods, namely, histograms of LBP (Local Binary Pattern) features [3] and Gabor Jet Descriptors with Borda count classifiers [46] are selected based on their performance in the studies reported in [36], [46]. The third local-matching method, histograms of WLD (Weber Linear Descriptor) features, recently proposed in [12], has shown very good performance in face detection applications, and is used here for the first time in face recognition. The SIFT (Scale-Invariant Feature Transform) image-matching method [26] is included following its good performance in a former face recognition study [36]. Finally, the SURF (Speeded Up Robust Features) image-matching method [5], which is inspired by the SIFT method, is included because in many applications in which the SIFT method is used, SURF obtains a similar performance and a higher speed.
The comparative study is carried out using the Equinox and UCHThermalFace databases. The Equinox database [16] was selected because it is one of the most frequently employed thermal face databases, and therefore it allows comparing the obtained results with former studies. The UCHThermalFace database was specially designed to study the problem of unconstrained face recognition in the thermal domain. The database incorporates thermal images acquired in indoor and outdoor setups, with natural variations in illumination, facial expression, pose, accessories, occlusions, and background. This database will be made public for future comparative studies, which is also a contribution of this paper.
This comparative study intends to be a complement to the recently published comparative study on visible-spectrum face recognition methods in unconstrained environments [36].
The paper is structured as follows: Related works are outlined in Section 2. The methods under analysis are described in Section 3. In 4 Comparative study using the equinox database, 5 Comparative study using the UCHThermalFace database the comparative analyses of these methods in the Equinox and UCHThermalFace databases are presented. Finally, in Section 6 results are discussed, and conclusions are given.
Section snippets
Related work
Several comparative studies of thermal face recognition approaches have been developed in recent years [37], [38], [40]. Most of the developed approaches make use of appearance-based methods, such as PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), and ICA (Independent Component Analysis), which project face images into a subspace where the recognition is carried out. These methods achieve a ∼95% recognition rate in experiments that do not consider real-world conditions
Methods under comparison
As mentioned above, the methods under comparison were selected considering their fulfillment of the defined requirements (real-time, fully online, just one image per person), and their performance in former comparative studies of face-recognition methods [35], [36], [41], [46] and in face detection applications [12].
Comparative study using the equinox database
The performance of the methods is analyzed using the Equinox database [16], which has already been used to validate several face recognition methods for thermal images. This allows the direct comparison of the selected methods with face recognition methods implemented in previous works: in [37], [38] methods based on LDA (Linear Discriminant Analysis), LFA (Local Feature Analysis), and ICA (Independent Component Analysis) have been compared. PCA (Principal Component Analysis) has been analyzed
Comparative study using the UCHThermalFace database
The methods under study are analyzed considering real-world conditions that include indoor/outdoor setups and natural variations on facial expression, pose, accessories, occlusions, and background.
Discussion and conclusions
In this article, a comparative study of thermal-based face-recognition methods in unconstrained environments was presented. The analyzed methods were selected by considering their suitability for the defined requirements —real-time operation, just one image per person, fully online (no training), and robust behavior in unconstrained environments—and their performance in former studies. The comparative study was carried out using two databases: Equinox and UCHThermalFace. The well-known Equinox
Acknowledgements
This research was partially funded by the FONDECYT-Chile Grant 1090250 and by the Advanced Mining Technology Center.
References (44)
- et al.
SURF: Speeded Up Robust Features
Computer Vision and Image Understanding (CVIU)
(2008) - et al.
Face recognition by fusing thermal infrared and visible imagery
Image Vision Computing
(2006) - et al.
Performance evaluation of score level fusion in multimodal biometric systems
Pattern Recognition
(2010) - et al.
Illumination compensation and normalization in eigenspace-based face recognition: a comparative study of different pre-processing approaches
Pattern Recognition Letters
(2008) - et al.
Face recognition from a single image per person: a survey
Pattern Recognition
(2006) - B. Abidi, S. Huq, M. Abidi, Fusion of visual, thermal, and range as a solution to illumination and pose restrictions in...
- J. Ahmad, U. Ali, R.J. Qureshi, Fusion of thermal and visual images for efficient face recognition using Gabor filter,...
- et al.
Face description with local binary patterns: application to face recognition,
IEEE Transactions on Pattern Analysis and Machine Intelligence
(2006) - M. Akhloufi, A. Bendada, Thermal faceprint: a new thermal face signature extraction for infrared face recognition, In...
- M.K. Bhowmik, D. Bhattacharjee, M. Nasipuri, D.K. Basu, M. Kundu, Optimum fusion of visual and thermal face images for...
Multi-spectral face recognition—fusion of visual imagery with physiological information
Face recognition beyond the visible spectrum, advances in biometrics: sensors
Algorithms and Systems
WLD: a robust local image descriptor
IEEE Transactions on Pattern Analysis and Machine Intelligence
Face databases
Cited by (89)
Attacks and defenses in user authentication systems: A survey
2021, Journal of Network and Computer ApplicationsAn extensive review on spectral imaging in biometric systems: Challenges & advancements
2019, Journal of Visual Communication and Image RepresentationSurvey on biometry for cognitive automotive systems
2019, Cognitive Systems ResearchFace presentation attack detection: Research opportunities and perspectives
2023, Intelligent Decision TechnologiesA comprehensive survey on techniques to handle face identity threats: challenges and opportunities
2023, Multimedia Tools and ApplicationsTowards a super-resolution based approach for improved face recognition in low resolution environment
2022, Multimedia Tools and Applications