Skip to main content
SpringerLink
Log in

Intensity matching through saliency maps for thermal and visible image registration for face detection applications

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Face identification is a common image processing problem that is used in a range of applications, including pedestrian tracking, population flow forecast, video surveillance, and fever screening. When combined with a visible-light camera, thermal imaging provides a more thorough image of the person. Face identification is commonly accomplished by combining data from many modalities, such as infrared(IR)/visible(VIS) pictures. The face detection system’s success is contingent upon the image registration approach being of good quality. In this article, we demonstrate an improved intensity-based image registration approach for visual and thermal face pictures. The approach applies a saliency map strategy to balance the infrared and visible pictures’ intensity levels in order to overcome intensity differences and ensure proper image registration. When applied to facial infrared and visible pictures, the proposed technique displays an improvement in registration quality when measured using the structural similarity index measure (SSIM) and universal image quality index (UQI) metrics. We report an average improvement of 16.93 % in terms of SSIM score and 7.02 % in terms of UQI score when the suggested image registration system is applied in comparison with unregistered images for the test IR/VIS face images. Furthermore, the proposed method outperformed the state-of-the-art Oriented FAST and Rotated BRIEF-based image registration methods in terms of image registration quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

Data will be made available on reasonable request to the corresponding author.

References

  1. Liu, C., Xu, J., Wang, F.: A review of keypoints’ detection and feature description in image registration. Sci. Program. 2021, 1–25 (2021)

    Google Scholar 

  2. Kim, J., Fessler, J.A.: Intensity-based image registration using robust correlation coefficients. IEEE Trans. Med. Imaging 23(11), 1430–1444 (2004)

    Article  Google Scholar 

  3. Li, M., Zhang, M., Niu, D., Hassan, M.U., Zhao, X., Li, N.: Point set registration based on feature point constraints. Vis. Comput. 36(9), 1725–1738 (2020). https://doi.org/10.1007/s00371-019-01771-x

    Article  Google Scholar 

  4. BiBi, S., Abbas, M., Miura, K.T., Misro, M.Y.: Geometric modeling of novel generalized hybrid trigonometric Bézier-like curve with shape parameters and its applications. Mathematics 8(6), 967 (2020). https://doi.org/10.3390/math8060967

    Article  Google Scholar 

  5. Maqsood, S., Abbas, M., Miura, K.T., Majeed, A., Iqbal, A.: Geometric modeling and applications of generalized blended trigonometric Bézier curves with shape parameters. Adv. Differ. Equ. 2020(1), 550 (2020). https://doi.org/10.1186/s13662-020-03001-4

    Article  MATH  Google Scholar 

  6. Dixit, M., Silakari, S.: Utility of parametric curves in image processing applications. Int. J. Signal Process. Image Process. Pattern Recognit. 8(7), 317–326 (2015)

    Google Scholar 

  7. Abbas, M., Ramli, N., Majid, A.A., Ali, J.M.: The representation of circular arc by using rational cubic timmer curve. Math. Probl. Eng. 2014, 1–6 (2014). https://doi.org/10.1155/2014/408492

    Article  MathSciNet  MATH  Google Scholar 

  8. Liu, F., Seipel, S.: Infrared-visible image registration for augmented reality-based thermographic building diagnostics. Vis. Eng. 3(1), 66 (2015)

    Article  Google Scholar 

  9. Saleem, S., Bais, A.: Visible spectrum and infra-red image matching: a new method. Appl. Sci. 10(3), 1162 (2020)

    Article  Google Scholar 

  10. Chen, Y., Zhang, X., Zhang, Y., Maybank, S.J., Fu, Z.: Visible and infrared image registration based on region features and edginess. Mach. Vis. Appl. 29(1), 113–123 (2018). https://doi.org/10.1007/s00138-017-0879-6

    Article  Google Scholar 

  11. Istenic, R., Heric, D., Ribaric, S., Zazula, D.: Thermal and visual image registration in hough parameter space. In: 2007 14th International Workshop on Systems, Signals and Image Processing and 6th EURASIP Conference Focused on Speech and Image Processing, Multimedia Communications and Services, pp. 106–109. IEEE; 2007. http://ieeexplore.ieee.org/document/4381164/

  12. Kumawat, A., Panda, S.: A robust edge detection algorithm based on feature-based image registration (FBIR) using improved canny with fuzzy logic (ICWFL). Vis. Comput. 6, 66 (2021). https://doi.org/10.1007/s00371-021-02196-1

    Article  Google Scholar 

  13. Kuppala, K., Banda, S., Barige, T.R.: An overview of deep learning methods for image registration with focus on feature-based approaches. Int. J. Image Data Fusion 11(2), 113–135 (2020). https://doi.org/10.1080/19479832.2019.1707720

    Article  Google Scholar 

  14. Fu, Y., Lei, Y., Wang, T., Curran, W.J., Liu, T., Yang, X.: Deep learning in medical image registration: a review. Phys. Med. Biol. 65(20), 20TR01 (2020). https://doi.org/10.1088/1361-6560/ab843e

    Article  Google Scholar 

  15. Shi, C., Zhang, J., Yao, Y., Sun, Y., Rao, H., Shu, X.: CAN-GAN: conditioned-attention normalized GAN for face age synthesis. Pattern Recognit. Lett. 138, 520–526 (2020). https://doi.org/10.1016/j.patrec.2020.08.021

    Article  Google Scholar 

  16. Raza, Se.A., Sanchez, V., Prince, G., Clarkson, J.P., Rajpoot, N.M.: Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain. Pattern Recognit. 48(7), 2119–2128 (2015). https://doi.org/10.1016/j.patcog.2015.01.027

    Article  Google Scholar 

  17. Yang, Y.: Non-rigid image registration for visible color and thermal IR face. In: 2016 International Conference on Audio, Language and Image Processing (ICALIP), pp. 279–284. IEEE. (2016). http://ieeexplore.ieee.org/document/7846571/

  18. Anzid, H., le Goic, G., Bekkari, A., Mansouri, A., Mammass, D.: A new SURF-based algorithm for robust registration of multimodal images data. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02435-z

    Article  Google Scholar 

  19. Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P.: Multimodality image registration by maximization of mutual information. IEEE Trans. Med. Imaging 16(2), 187–198 (1997). https://doi.org/10.1109/42.563664

    Article  Google Scholar 

  20. D’Agostino, E., Maes, F., Vandermeulen, D., Suetens, P.: A viscous fluid model for multimodal non-rigid image registration using mutual information. Med. Image Anal. 7(4), 565–575 (2003)

    Article  MATH  Google Scholar 

  21. Krishnan, P.T., Balasubramanian, P., Jeyakumar, V.: Histogram matched visible and infrared image registration for face detection. In: IEEE EUROCON 2021—19th International Conference on Smart Technologies, pp. 222–226. IEEE (2021)

  22. Zhang, X., Ma, Y., Fan, F., Zhang, Y., Huang, J.: Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition. J. Opt. Soc. Am. A Opt. Image Sci Vis. 34(8), 1400 (2017)

    Article  Google Scholar 

  23. Guo, B., Zhang, Q., Hou, Y.: Region-based fusion of infrared and visible images using nonsubsampled contourlet transform. Chin. Opt. Lett. 6(5), 338–341 (2008)

    Article  Google Scholar 

  24. Ali, F.E., El-Dokany, I.M., Saad, A.A., Al-Nuaimy, W., Abd El-Samie, F.E.: High resolution image acquisition from magnetic resonance and computed tomography scans using the curvelet fusion algorithm with inverse interpolation techniques. Appl. Opt. 49(1), 114–125 (2010)

    Article  Google Scholar 

  25. Bhardwaj, J., Nayak, A.: Lifting wavelet and KL transform (LWKL) based CT and MRI image fusion scheme. In: Biophotonics Congress: Optics in the Life Sciences Congress 2019 (BODA, BRAIN, NTM, OMA, OMP), p. JT4A.5. OSA, Washington, DC (2019)

  26. Guo, H., Ma, Y., Mei, X., Ma, J.: Infrared and visible image fusion based on total variation and augmented Lagrangian. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 34(11), 1961–1968 (2017)

    Article  Google Scholar 

  27. Foroosh, H., Zerubia, J.B., Berthod, M.: Extension of phase correlation to subpixel registration. IEEE Trans. Image Process. 11(3), 188–200 (2002)

    Article  Google Scholar 

  28. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2. pp. 1150–1157. IEEE (1999)

  29. Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L., Leach, M.O., Hawkes, D.J.: Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans. Med. Imaging 18(8), 712–721 (1999)

    Article  Google Scholar 

  30. Tang, T.W.H., Chung, A.C.S.: Non-rigid image registration using graph-cuts. Med. Image Comput. Assist. Interv. 10(Pt 1), 916–924 (2007)

    Google Scholar 

  31. So, R.W.K., Tang, T.W.H., Chung, A.C.S.: Non-rigid image registration of brain magnetic resonance images using graph-cuts. Pattern Recognit. 44(10–11), 2450–2467 (2011)

    Article  Google Scholar 

  32. Chen, K., Grapiglia, G.N., Yuan, J., Zhang, D.: Improved optimization methods for image registration problems. Numer. Algorithms 80(2), 305–336 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  33. Wan, X., Wang, C., Li, S.: The extension of phase correlation to image perspective distortions based on particle swarm optimization. Sensors 19(14), 3117 (2019)

    Article  Google Scholar 

  34. Shu, X., Tang, J., Li, Z., Lai, H., Zhang, L., Yan, S.: Personalized age progression with bi-level aging dictionary learning. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 905–917 (2018). https://doi.org/10.1109/TPAMI.2017.2705122

    Article  Google Scholar 

  35. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: 2011 International Conference on Computer Vision, pp. 2564–2571 (2011)

  36. Shu, X., Tang, J., Lai, H., Liu, L., Yan, S.: Personalized age progression with aging dictionary. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3970–3978 (2015). IEEE. http://ieeexplore.ieee.org/document/7410809/

  37. The tufts face database. Accessed 2021-5-13. http://tdface.ece.tufts.edu/

  38. Panetta, K., Wan, Q., Agaian, S., Rajeev, S., Kamath, S., Rajendran, R., et al.: A comprehensive database for benchmarking imaging systems. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 509–520 (2020). https://doi.org/10.1109/TPAMI.2018.2884458

    Article  Google Scholar 

  39. Hou, X., Zhang, L.: Saliency detection: a spectral residual approach. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

  40. Artyushkova, K.: Automatic image registration using (normalized) mutual information for users of IP toolbox—File Exchange—MATLAB Central. Accessed 2021-5-14. https://in.mathworks.com/matlabcentral/fileexchange/4145-automatic-image-registration-using-normalized-mutual-informationfor-users-of-ip-toolbox

  41. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN’95 - International Conference on Neural Networks. IEEE (2002)

  42. Ye, H., Luo, W., Li, Z.: Convergence analysis of particle swarm optimizer and its improved algorithm based on velocity differential evolution. Comput. Intell. Neurosci. 2013, 384125 (2013)

    Article  Google Scholar 

  43. Qian, W., Li, M.: Convergence analysis of standard particle swarm optimization algorithm and its improvement. Soft Comput. 22(12), 4047–4070 (2018)

    Article  MATH  Google Scholar 

  44. Wang, Z., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)

    Article  Google Scholar 

  45. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Science and Engineering Research Board (SERB), Department of Science and Technology, India, through the Core Research Grant (File No. CRG/2020/003042).

Author information

Authors and Affiliations

  1. Department of Electronics and Instrumentation Engineering, St.Joseph’s College of Engineering, Anna University, Chennai, Tamilnadu, 600119, India

    Palani Thanaraj Krishnan

  2. Department of Computer Science and Engineering, St.Joseph’s College of Engineering, Anna University, Chennai, Tamilnadu, 600119, India

    Parvathavarthini Balasubramanian

  3. Department of Biomedical Engineering, Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, Tamilnadu, 603110, India

    Vijay Jeyakumar

  4. Department of Endocrinology, Sri Ramachandra Medical College and Research Institute, Porur, Tamilnadu, 600116, India

    Shriraam Mahadevan

  5. Department of Electronic Engineering, Shantou University, Shantou, 515063, Guangdong Province, China

    Alex Noel Joseph Raj

Authors
  1. Palani Thanaraj Krishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Parvathavarthini Balasubramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vijay Jeyakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shriraam Mahadevan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alex Noel Joseph Raj
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

PTK provided conceptualization, methodology, and writing. PB did supervision, review, and editing. VJ contributed to formal analysis and investigation. SM performed validation and data curation. ANJR did visualization and software.

Corresponding author

Correspondence to Vijay Jeyakumar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (rar 119 KB)

Supplementary file 2 (rar 778 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Krishnan, P.T., Balasubramanian, P., Jeyakumar, V. et al. Intensity matching through saliency maps for thermal and visible image registration for face detection applications. Vis Comput 39, 4529–4542 (2023). https://doi.org/10.1007/s00371-022-02605-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02605-z

Keywords

Search

Navigation