Skip to main content
SpringerLink
Log in

Bridging the spectral gap using image synthesis: a study on matching visible to passive infrared face images

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

We propose an approach that bridges the gap between the visible and IR band of the electromagnetic spectrum, namely the mid-wave infrared or MWIR (3–5 \(\upmu \hbox {m}\)) and the long-wave infrared or LWIR (8–14 \(\upmu \hbox {m}\)) bands. Specifically, we investigate the benefits and limitations of using synthesized visible face images from thermal and vice versa, in cross-spectral face recognition systems when utilizing canonical correlation analysis and manifold learning dimensionality reduction. There are four primary contributions of this work. First, we assemble a database of frontal face images composed of paired VIS-MWIR and VIS-LWIR face images (using different methods for pre-processing and registration). Second, we formulate a image synthesis framework and post-synthesis restoration methodology, to improve face recognition accuracy. Third, we explore cohort-specific matching (per gender) instead of blind-based matching (when all images in the gallery are matched against all in the probe set). Finally, by conducting an extensive experimental study, we establish that the proposed scheme increases system performance in terms of rank-1 identification rate. Experimental results suggest that matching visible images against images acquired with passive infrared spectrum, and vice-versa, are feasible with promising results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Ansari, A., Mahoor, M., Abdel-Mottaleb, M.: Normalized 3D to 2D model-based facial image synthesis for 2D model-based face recognition. In: IEEE GCC Conference and Exhibition (GCC), pp. 178–181 (2011)

  2. Belhumeur, P., Hespanha, J., Kriegman, D.J.: Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 711–720 (1997)

    Article  Google Scholar 

  3. Bolme, D., Beveridge, J., Teixeira, M., Draper, B.: The CSU face identification evaluation system: its purpose, features and structure. In: Proceedings of International Conference on Vision Systems, pp. 301–311 (2003)

  4. Bourlai, T., Kalka, N., Cao, D., Decann, B., Jafri, Z., Nicolo, F., Whitelam, C., Zuo, J., Adjeroh, D., Cukic, B., Dawson, J., Hornak, L., Ross, A., Schmid, N.A.: Ascertaining Human Identity in Night Environments. Princeton University Press, Princeton (2010a)

    Google Scholar 

  5. Bourlai, T., Kalka, N., Ross, A., Cukic, B., Hornak, L.: Cross-spectral face verification in the short wave infrared (SWIR) band (2010b)

  6. Bourlai, T., Ross, A., Chen, C., Hornak, L.: A study on using middle-wave infrared images for face recognition. In: SPIE, Biometric Tech for Human Identification IX (2012)

  7. Byrd, K.: Preview of the newly acquired NVESD-ARL multimodal face database. In: Proceedings of SPIE, vol. 8734, p. 34 (2013)

  8. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 886–893 (2005)

  9. Devijver, A.P., Kittler, J.: Pattern Recognition: A Statistical Approach. Prentice-Hall, London (1982)

    MATH  Google Scholar 

  10. Dou, M., Zhang, C., Hao, P., Li, J.: Converting thermal infrared face images into normal gray-level images. In: ACCV (2007)

  11. Chang, H., Yeung, D., Xiong, Y.: Super-resolution through neighbor embedding. In: CVPR (2004)

  12. Klare, B., Jain, A.: Heterogeneous face recognition: matching NIR to visible light images. In: ICPR, pp. 1513–1516 (2010)

  13. Klare, B., Jain, A.: Heterogeneous face recognition using kernel prototype similarities. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1410–1422 (2013)

    Article  Google Scholar 

  14. Klare, B., Li, Z., Jain, A.: Matching forensic sketches to mug shot photos. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 639–646 (2011)

    Article  Google Scholar 

  15. Lei, Z., Li, S.: Coupled spectral regression for matching heterogeneous faces. In: CVPR (2009)

  16. Lei, Z., Liao, S., Jain, A., Li, S.: Coupled discriminant analysis for heterogeneous face recognition. IEEE Trans. Inf. Forensics Secur. 7(6), 1707–1716 (2012)

    Article  Google Scholar 

  17. Li, C., Su, G., Shang, Y., Li, Y., Xiang, Y.: Face recognition based on pose-variant image synthesis and multi-level multi-feature fusion. In: AMFG, pp. 261–275 (2007)

  18. Lin, D., Tang, X.: Inter-modality face recognition. Proc. Eur. Conf. Comput. Vis. 3954, 13–26 (2006)

    Google Scholar 

  19. Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: CVPR, vol. 1, pp. 1005–1010 (2005)

  20. Liu, W., Liu, J., Tang, X.: Bayesian tensor inference for sketch-based facial photo hallucination. In: IJCAI, pp. 2141–2146 (2007)

  21. Lu, X., Hsu, R., Jain, A., Kamgar-Parsii, B.: Face recognition with 3D model-based synthesis. In: Proceedings of International Conference on Biometric Authentication (ICBA), pp. 139–146 (2004)

  22. Majumdar, A., Ward, R.K.: Single image per person face recognition with images synthesized by non-linear approximation. In: International Conference on Image Processing, pp. 2740–2743 (2008)

  23. Melzer, T., Reiter, M., Bischof, H.: Appearance models based on kernel canonical correlation analysis. Pattern Recognit. 36, 1961–1971 (2003)

    Article  MATH  Google Scholar 

  24. Mohideen, S.K., Perumal, S.A., Sathik, M.M.: Image de-noising using discrete wavelet transform. Int. J. Comput. Sci. Netw. Secur. 8(1), 213–216 (2008)

    Google Scholar 

  25. Muller, K., Mika, S., Ratsch, G., Tsuda, K., Scholkopf, B.: An introduction to kernel-based learning algorithms. In: IEEE Transactions on Neural Networks, pp. 181–201 (2001)

  26. Pietikinen, M.: Image analysis with local binary patterns. In: Proceedings of Scandinavian Conference on Image Analysis, pp. 115–118 (2005)

  27. Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)

    Article  Google Scholar 

  28. Sarfraz, M,, Stiefelhagen, R.: Deep perceptual mapping for thermal to visible face recognition. In: British Machine Vision Conference (BMVC) (2016)

  29. Selinger, A., Socolinksy, D.A.: Face recognition in the dark. In: CPRW, pp. 129–134 (2004)

  30. Shuowen, H., Short, N., Gurram, P., Gurton, K., Reale, C.: FR Across the Imaging Spectrum, chap. 4 (2016)

  31. Sirovich, L., Kirby, M.: Application of the Karhunen-Loeve procedure for the characterization of human faces. IEEE Trans. Pattern Anal. Mach. Intell. 12(1), 103–108 (1990)

  32. Socolinsky, D., Selinger, A., Neuheisel, J.: Face recognition with visible and thermal imagery. CVIU 91, 72–114 (2003)

    Google Scholar 

  33. Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. Trans. Image Process. 19, 1635–1650 (2010)

    Article  MathSciNet  Google Scholar 

  34. Tang, X., Wang, X.: Face sketch recognition. IEEE Trans. Circuits Syst. Video Technol. 14(1), 50–57 (2004)

    Article  Google Scholar 

  35. Teixeira, M.: The bayesian intrapersonal/extrapersonal classifier. Ph.D. thesis, Colorado State University (2003)

  36. Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cogn. Neurosci. 3(1), 71–86 (1991)

    Article  Google Scholar 

  37. Viola, P., Jones, M.: Robust real-time face detection. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  38. Wang, R., Yang, J., Yi, D., Li, S.: An analysis-by-synthesis method for heterogeneous face biometrics. In: ICB (2009)

  39. Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009)

    Article  Google Scholar 

  40. Wilder, J., Phillips, P., Jiang, C., Wiener, S.: Comparison of visible and infra-red imagery for face recognition. In: Automatic Face and Gesture Recognition, pp. 182–187 (1996)

  41. Yi, D., Liu, R., Chu, R., Lei, Z., Li, S.: Partial face matching between near infrared and visual images in MBGC portal challenge. In: ICB (2007)

  42. Yi, D., Liao, S., Lei, Z., Sang, J., Li, SZ.: Partial face matching between near infrared and visual images in MBGC portal challenge. In: ICB, Springer, pp. 733–742 (2009)

  43. Yoshitomi, Y., Miyaura, T., Tomita, S., Kimura, S.: Face identification using thermal image processing. In: WRHC, pp. 374–379 (1997)

  44. Zhang, W., Wang, X., Tang, X.: Coupled information-theoretic encoding for face photo-sketch recognition. In: CVPR, pp. 513–520 (2011a)

  45. Zhang, Z., Wang, Y., Zhang, Z.: Face synthesis from near-infrared to visual light via sparse representation. In: ICB (2011b)

Download references

Acknowledgements

The authors would like to thank Dr. Mingsong Dou for his contributions in helping understand important concepts from the initial study [10]. The authors would also like to thank Dr. Shuowen Hu and the US Army Research Laboratory for granting us access to the NVESD dataset used in this work.

Author information

Authors and Affiliations

  1. MILab LCSEE, West Virginia University, 395 Evansdale Drive, Morgantown, WV, 26506, USA

    Nnamdi Osia & Thirimachos Bourlai

Authors
  1. Nnamdi Osia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thirimachos Bourlai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nnamdi Osia.

Additional information

This material is based upon work supported by the Center for Identification Technology Research and the National Science Foundation under Grant No. 1066197.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Osia, N., Bourlai, T. Bridging the spectral gap using image synthesis: a study on matching visible to passive infrared face images. Machine Vision and Applications 28, 649–663 (2017). https://doi.org/10.1007/s00138-017-0855-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-017-0855-1

Keywords

Search

Navigation