Skip to main content
SpringerLink
Log in

Optimal fusion aided face recognition from visible and thermal face images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a novel Eigen face recognition that is aided by fusion of visible and thermal face images to improve the face recognition accuracy. We adopt three different fusion schemes where in the face information is fused by the optimal weights obtained by different optimization algorithms. The first two fusion approaches operate in the dual tree discrete wavelet transform (DT-DWT), while the third one operates in the Curvelet transform (CT) domain. We employ particle swarm optimization (PSO), self-tuning particle swarm optimization (STPSO) and brain storm optimization algorithm (BSO) to find optimal fusion coefficients. The proposed fusion aided face recognition approaches are evaluated through extensive experiments using OCTVBS benchmark face database and the Eigen face detection methodology. Simulation results show that proposed face recognition techniques have significant performance improvement in recognition accuracy suggesting fusion aided face recognition approach that deserves further study and consideration whenever high recognition accuracy is desired.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig 10
Fig. 11
Fig. 12
Fig 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Ahmad A, Riaz MM, Ghafoor A, Zaidi T (2016) Noise resistant fusion for multi-exposure sensors. IEEE Sensors J 16(13):5123–5124

    Google Scholar 

  2. Arivazhagan S, Mumtaj J (2007) Face Recognition using Multi-Resolution Transform. International Conference on Computational Intelligence and Multimedia Applications, 301–306

  3. Bebis G, Gyaourova A, Singh S, Pavlidis I (2006) Face recognition by fusing thermal infrared and visible imagery. Image Vis Comput 24:727–742

    Google Scholar 

  4. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs.fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Google Scholar 

  5. BenAbdelkader C, Griffin P (2005) A local region-based approach to gender classification from face images. Proc. IEEE Comp. Society Conf. Computer Vision and Pattern Recognition, pages 52–57

  6. Brunelli R, Poggio T (1993) Face recognition: features versus templates. IEEE Trans Pattern Anal Mach Intell 15(10):1042–1052

    Google Scholar 

  7. Chen W, Er MJ, Wu S (2006) “Illumination compensation and normalization for robust face recognition using discrete cosine transform in logarithm domain” IEEE transactions on systems, man, and cybernetics. Part B (Cybernetics) 36(2):458–466

    Google Scholar 

  8. Chen YL, Jahanshahi MR, Manjunatha P, Gan WP, Abdelbarr M, Masri SF, Becerik-Gerber B, Caffrey JP (2016) Inexpensive multimodal sensor fusion system for autonomous data acquisition of road surface conditions. IEEE Sensors J 16(21):7731–7743

    Google Scholar 

  9. Choudhary A, Vig R (2018) Face Recognition Using Multiresolution Hybrid Kekre-DCT Wavelet Transform Features with Multiclass ECOC Framework. Proc Comp Sci 132:1781–1787

    Google Scholar 

  10. Choudhary A, Vig R (2018) Face recognition using multiresolution hybrid Kekre-DCT wavelet transform features with multiclass ECOC framework. Proc Comp Sci 132:1781–1787

    Google Scholar 

  11. Dakin SC, Watt RJ (2009) Biological `bar codes' in human faces. World Acad Sci Eng Technol 9:1–10

    Google Scholar 

  12. Ekenel HK, Sankur B (2005) Multiresolution face recognition. Image Vis Comput 23:469–477

    Google Scholar 

  13. Eleyan A, Demirel H (2007) Face Recognition using Multiresolution PCA. IEEE International Symposium on Signal Processing and Information Technology, 52–55.

  14. Er MJ, Chen W, Wu S (2005) High-speed face recognition based on discrete cosine transform and RBF neural networks. IEEE Trans Neural Netw 16(3):679–691

    Google Scholar 

  15. Gao Y, Ma J (2017) Semi-Supervised Sparse Representation Based Classification for Face Recognition with Insufficient Labeled Samples. IEEE Transactions on Image Processing

  16. Ghasemzadeh A, Demirel H (2017) 3D discrete wavelet transform-based feature extraction for hyperspectral face recognition. IET Biometrics 7(1):49–55

    Google Scholar 

  17. Grotschel M, Lovász L (1993) Combinatorial optimization: a survey, DIMACS technical report 93–29, Princeton University. In Internet

  18. Guzman AM, Goryawala M, Wang J, Barreto A, Andrian J, Rishe N, Adjouadi M (2013) Thermal imaging as a biometrics approach to facial signature authentication. IEEE J Biomed Health Inf 17(1):214–222

    Google Scholar 

  19. Hermosilla G, Gallardo F, Farias G, San Martin C (2015) Fusion of visible and thermal descriptors using genetic algorithms for face recognition systems. Sensors 15:17944–17962. https://doi.org/10.3390/s150817944

    Article  Google Scholar 

  20. Hizem W, Allano L, Mellakh A, Dorizzi B (2009) Face recognition from synchronised visible and near-infrared images. IET Signal Process 3(4):282–288

    Google Scholar 

  21. Hollingsworth KP, Darnell SS, Miller PE, Woodard DL, Bowyer KW, Flynn PJ (2012) Human and Machine Performance on Periocular Biometrics Under Near-Infrared Light and Visible Light. IEEE Transact Inf Foren Sec 7(2):588–601

    Google Scholar 

  22. IEEE OTCBVS WS Series Bench (n.d.); DOE University Research Program in Robotics under grant DOE-DE-FG02-86NE37968; DOD/TACOM/NAC/ARC Program under grant R01–1344-18; FAA/NSSA grant R01–1344-48/49; Office of Naval Research under grant #N000143010022

  23. IEEE OTCBVS WS Series Bench (n.d.); DOE University Research Program in Robotics under grant DOE-DE-FG02-86NE37968; DOD/TACOM/NAC/ARC Program under grant R01–1344-18; FAA/NSSA grant R01–1344-48/49; Office of Naval Research under grant #N000143010022. http://vcipl-okstate.org/pbvs/bench/Data/02/download.html

  24. Imtiaz H, Fattah SA (2011) A face recognition scheme using waveletbased dominant features. Signal & Image Processing : An International Journal (SIPIJ). 2(3)

  25. Kim T-K, Kittler J (2005) Locally Linear Discriminant Analysis for Multi-modally Distributed Classes for Face Recognition with a Single Model Image. IEEE Trans Pattern Anal Mach Intell 27(3):318–327

    Google Scholar 

  26. Klare BF, Jain AK Heterogeneous Face Recognition Using Kernel Prototype Similarities. IEEE Trans Pattern Anal Mach Intell 35(6):1410–1422

  27. Kong SG, Heo J, Boughorbel F, Zheng Y, Abidi BR, Koschan A, Yi M, Abidi MA (2007) Multiscale Fusion of Visible and Thermal IR Images for Illumination-Invariant Face Recognition. Int J Comput Vis 71(2):215–233

    Google Scholar 

  28. Kong WW, Lei YJ, Lei Y, Zhang J (2010) Technique for image fusion based on non-subsampled contourlet transform domain improved NMF. SCIENCE CHINA Inf Sci 53(12):2429–2440

    MathSciNet  MATH  Google Scholar 

  29. Kumar A, Zhang D (2006) Personal recognition using hand shape and texture. IEEE Trans Image Process 15(8):2454–2461

    Google Scholar 

  30. Laurenz W, Fellous JM, Krüger N, von der Malsburg C (1997) Face recognition by elastic bunch graph matching. 19(7): 775–779

  31. Lawrence S, Giles CL, Tsoi A, Back AD (1998) Face recognition: a convolutional neural network approach. IEEE Trans Neural Netw 8(1):98–113

    Google Scholar 

  32. Li SZ, Chu RF, Liao SC, Zhang L (2007) Illumination invariant face recognition using near-infrared images. IEEE Trans Pattern Anal Mach Intell 29(4):627–639

    Google Scholar 

  33. Liu C (2004) Gabor-Based Kernel PCA with Fractional Power Polynomial Models for Face Recognition. IEEE Trans Pattern Anal Mach Intell 26(5):572–581

    Google Scholar 

  34. Ma J, Zhao J, Ma Y, Tian J (2015) Non-rigid visible and infrared face registration via regularized Gaussian fields criterion. Pattern Recogn 48:772–784

    Google Scholar 

  35. Ma J, Qiu W, Zhao J, Ma Y, Yuille AL (2015) Robust L2E Estimation of Transformation for Non-Rigid Registration, IEEE Transactions on Signal Processing. IEEE Transactions on Signal Processing, VOL. 63, NO. 5, MARCH 1, 2015, 1115–1129

  36. Ma J, Chen C, Li C, Huang J (2016) Infrared and visible image fusion via gradient transfer and total variation minimization. Info Fusion 31:100–109

    Google Scholar 

  37. Ma J, Ma Y, Li C (2019) Infrared and visible image fusion methods and applications: a survey. Info Fusion 45:153–178

    Google Scholar 

  38. Madheswari K, Venkateswaran N (2016) Swarm intelligence based optimisation in thermal image fusion using dual tree discrete wavelet transform. Quant Infra Thermograph J, First Online, 1–20

  39. MartõÂnez AM, Kak AC (2001) PCA versus LDA. IEEE Trans Pattern Anal Mach Intell 23(2):228–233

    Google Scholar 

  40. Meraoumia A, Chitroub S, Bouridane A (2010) Gaussian modeling and Discrete Cosine Transform for efficient and automatic palmprint identification. Int Conf Mach Web Intell (ICMWI):121–125

  41. Moghaddam B, Pentland A (1997) Probabilistic visual learning for object representation. IEEE Trans Pattern Analysis Mach Intell 18:696–710

    Google Scholar 

  42. Nicolò F, Schmid NA (2012) Long range cross-spectral face recognition: matching SWIR against visible light images. IEEE Transact Inf Foren Sec 7:1717–1725

    Google Scholar 

  43. Osuna E, Freund R, Girosi F (1997) Training support vector machines: an application to face detection

  44. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep Face Recognition British Machine Vision Conference

  45. Penev P, Atick J (1996) Local Feature Analysis: A general statistical theory for object representation

  46. Petrovic V (2007) Subjective tests for image fusion evaluation and objective metric validation. Info Fusion 8(2):2018–2216

    Google Scholar 

  47. Raghavendra R, Dorizzi B, Rao A, Kumar GH (2011) Particle swarm optimization based fusion of near infrared and visible images for improved face verification. Pattern Recogn 44:401–411

    Google Scholar 

  48. Rajoub BA, Zwiggelaar R (2014) Thermal facial analysis for deception detection. IEEE Transact Inf Foren Sec 9(6):1015–1023

    Google Scholar 

  49. Schoelkopf B, Smola A, Muller KR (1997) Kernel principal components analysis”, artificial neural networks, ICANN97

  50. Seal A, Bhattacharjee D, Nasipuri M (2016) Human face recognition using random forest based fusion of à-trous wavelet transform coefficients from thermal and visible images. Int J Electron Commun (AEÜ) 70:1041–1049

    Google Scholar 

  51. Shen LL, Bai L (2004) Gabor feature based face recognition using kernal methods. Proc IEEE Int Conf Auto Face Gest Recog:386–389

  52. Tan H, Huang X, Tan H, He C Pixel-level Image Fusion Algorithm Based on Maximum Likelihood and Laplacian Pyramid Transformation. J Comput Inf Syst 9(1):327–334

  53. Toet A (1989) Image fusion by a ratio of low-pass pyramid. Pattern Recogn Lett 9:245–253

    MATH  Google Scholar 

  54. Tripathi BK (2017) On the complex domain deep machine learning for face recognition. Appl Intell 47(2):382–396

    Google Scholar 

  55. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Sci:71–86

  56. Turk M, Pentland A (1991) Eigen faces for recognition. IEEE Transact Cogn Neurosci 13(1):71–86

    Google Scholar 

  57. Turk MA, Pentland AP (1991) Face recognition using eigenfaces. Computer Vision and Pattern Recognition. Proceedings {CVPR'91.}, {IEEE} Computer Society Conference on 1991

  58. Villegas-Quezada C, Climent J (2008) Holistic face recognition using multivariate approximation,genetic algorithms and AdaBoost classifier: preliminary results. World Acad Sci Eng Technol 44:802–806

    Google Scholar 

  59. Xiaozheng Z, Gao Y (2009) Face recognition across pose: A review. Pattern Recogn 42:2876–2896

    Google Scholar 

  60. Yang Y, Que Y (2016) Shuying Huang, and Pan Lin, ‘multimodal sensor medical image fusion based on Type-2 fuzzy logic in NSCT domain’. IEEE Sensors J 16(10):3735–3745

    Google Scholar 

  61. Yang J, Zhang D, Frangi AF, Yang J-y (2004) Two-Dimensional PCA : A New Approach to Appearance-Based Face Representation and Recognition. IEEE Trans Pattern Anal Mach Intell 26(1):131–137

    Google Scholar 

  62. Zheng L, Blasch E, Xue Z, Zhao J, Laganiere R, Wei W (2012) Objective Assessment of Multi-resolution Image Fusion algorithms for Context enhancement in Night Vision: A Comparative Study. IEEE Trans Pattern Anal Mach Intell 34(1):94–109

    Google Scholar 

  63. Zheng L, Blasch E, ZhiyunXue JZ, Laganiere R, Wu W (2012) Objective assessment of multi-resolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Trans Pattern Anal Mach Intell 34(1)

  64. Zhou M, Wei H (2006) Face verification using Gabor wavelets and adaboost. Proc Int Conf Pattern Recog:404–407

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering, SSN College of Engineering, Chennai, Tamil Nadu, India

    Madheswari Kanmani

  2. Electronics and Communication Engineering, SSN College of Engineering, Chennai, Tamil Nadu, India

    Venkateswaran Narasimhan

Authors
  1. Madheswari Kanmani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Venkateswaran Narasimhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Madheswari Kanmani.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kanmani, M., Narasimhan, V. Optimal fusion aided face recognition from visible and thermal face images. Multimed Tools Appl 79, 17859–17883 (2020). https://doi.org/10.1007/s11042-020-08628-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08628-9

Keywords

Search

Navigation