Abstract
We propose a multi-scale thermal-to-visible face synthesis system to achieve thermal face recognition. A generative adversarial network is constructed by one generator that transforms a given thermal face into a face in the visible spectrum, and three discriminators that consider multi-scale feature matching and high-frequency components, respectively. In addition, we provide a new paired thermal-visible face dataset called VTF that mainly contains Asian subjects captured in various visual conditions. This new dataset not only poses technical challenges to thermal face recognition, but also enables us to point out the race bias issue in current thermal face recognition methods. Overall, the proposed system achieves the state-of-the-art performance in both the EURECOM and NCKU-VTF datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? In: Proceedings of International Conference on Computer Vision, pp. 1021–1030 (2017)
Chatterjee, S., Chu, W.T.: Thermal face recognition based on transformation by residual U-net and pixel shuffle upsampling. In: Proceedings of International Conference on Multimedia Modelling (2020)
Chen, X., Flynn, P.J., Bowyer, K.W.: IR and visible light face recognition. Comput. Vis. Image Underst. 99(3), 332–358 (2005)
Chu, W.T., Huang, P.S.: Thermal face recognition based on multi-scale image synthesis. In: Proceedings of International Conference on Multimedia Modelling (2021)
Chu, W.T., Wu, J.N.: A parametric study of deep perceptual model on visible to thermal face recognition. In: Proceedings of IEEE International Conference on Visual Communications and Image Processing (2018)
Di, X., Riggan, B.S., Hu, S., Short, N.J., Patel, V.M.: Polarimetric thermal to visible face verification via self-attention guided synthesis. In: Proceedings of IAPR International Conference on Biometrics (2019)
Gao, Y., et al.: High-fidelity and arbitrary face editing. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 87–102. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_6
Isola, P., Zhu, J., Zhou, T., Efros, A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kresnaraman, B., Deguchi, D., Takahashi, T., Mekada, Y., Ide, I., Murase, H.: Reconstructing face image from the thermal infrared spectrum to the visible spectrum. Sensors 16(4), 568 (2016)
Mallat, K., Damer, N., Boutros, F., Kuijper, A., Dugelay, J.: Cross-spectrum thermal to visible face recognition based on cascaded image synthesis. In: Proceedings of International Conference on Biometrics (2019)
Mallat, K., Dugelay, J.: A benchmark database of visible and thermal paired face images across multiple variations. In: Proceedings of International Conference of the Biometrics Special Interest Group (2018)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Sarfraz, M.S., Stiefelhagen, R.: Deep perceptual mapping for thermal to visible face recognition. In: Proceedings of British Machine Vision Conference (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations (2015)
Wang, J., et al.: Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
Wang, T., Liu, M., Zhu, J., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8798–8807 (2018)
Wu, X., He, R., Sun, Z., Tan, T.: A light CNN for deep face representation with noisy labels. IEEE Trans. Inf. Forensics Secur. 13(11), 2884–2896 (2018)
Yi, D., Lei, Z., Liao, S., Li, S.: Learning face representation from scratch (2014). arXiv:1411.7923
Zhang, T., Wiliem, A., Yang, S., Lovell, B.: TV-GAN: generative adversarial network based thermal to visible face recognition. In: Proceedings of International Conference on Biometrics (2018)
Acknowledgement
This work was funded in part by Qualcomm through a Taiwan University Research Collaboration Project and in part by the National Science and Technology Council, Taiwan, under grants 111-3114-8-006-002, 110-2221-E-006-127-MY3, 108-2221-E-006-227-MY3, 107-2923-E-006-009-MY3, and 110-2634-F-006-022.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ho, TH., Yu, CY., Ko, TY., Chu, WT. (2023). The NCKU-VTF Dataset and a Multi-scale Thermal-to-Visible Face Synthesis System. In: Dang-Nguyen, DT., et al. MultiMedia Modeling. MMM 2023. Lecture Notes in Computer Science, vol 13833. Springer, Cham. https://doi.org/10.1007/978-3-031-27077-2_36
Download citation
DOI: https://doi.org/10.1007/978-3-031-27077-2_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27076-5
Online ISBN: 978-3-031-27077-2
eBook Packages: Computer ScienceComputer Science (R0)