Skip to main content
SpringerLink
Log in

A novel deep network architecture for reconstructing RGB facial images from thermal for face recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This work proposes a fully convolutional network architecture for RGB face image generation from a given input thermal face image to be applied in face recognition scenarios. The proposed method is based on the FusionNet architecture and increases robustness against overfitting using dropout after bridge connections, randomised leaky ReLUs (RReLUs), and orthogonal regularization. Furthermore, we propose to use a decoding block with resize convolution instead of transposed convolution to improve final RGB face image generation. To validate our proposed network architecture, we train a face classifier and compare its face recognition rate on the reconstructed RGB images from the proposed architecture, to those when reconstructing images with the original FusionNet, as well as when using the original RGB images. As a result, we are introducing a new architecture which leads to a more accurate network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Aitken A, Ledig C, Theis L, Caballero J, Wang Z, Shi W (2017) Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize, arXiv:1707.02937

  2. Anbarjafari G, Demirel H (2011) Modern: Face recognition. VDM Publishing

  3. Anbarjafari G (2013) Face recognition using color local binary pattern from mutually independent color channels. EURASIP J Image Video Process 2013(1):6

    Article  Google Scholar 

  4. Anbarjafari G, Haamer RE, Lusi I, Tikk T, Valgma L (2018) 3D face reconstruction with region based best fit blending using mobile phone for virtual reality based social media. Bulletin of the Polish Academy of Sciences Technical Sciences

  5. Bebis G, Gyaourova A, Singh S, Pavlidis I (2006) Face recognition by fusing thermal infrared and visible imagery. Image Vis Comput 24(7):727–742

    Article  Google Scholar 

  6. Bourlai T, Hornak LA (2016) Face recognition outside the visible spectrum. Image Vis Comput 55:14–17

    Article  Google Scholar 

  7. Brock A, Lim T, Ritchie JM, Weston N (2016) Neural photo editing with introspective adversarial networks, arXiv:1609.07093

  8. Buddharaju P, Pavlidis IT, Tsiamyrtzis P, Bazakos M (2007) Physiology-based face recognition in the thermal infrared spectrum. IEEE Trans Pattern Anal Mach Intell 29(4):613–626

    Article  Google Scholar 

  9. Daneshmand M, Helmi A, Avots E, Noroozi F, Alisinanoglu F, Arslan HS, Gorbova J, Haamer RE, Ozcinar C, Anbarjafari G (2018) 3D Scanning: A comprehensive survey, arXiv:1801.08863

  10. Demirel H, Anbarjafari G (2008) Pose invariant face recognition using probability distribution functions in different color channels. IEEE Signal Process Lett 15:537–540

    Article  Google Scholar 

  11. Demirel H, Anbarjafari G, Jahromi MNS (2008) Image equalization based on singular value decomposition. In: 2008. ISCIS’08. 23rd International Symposium on Computer and Information Sciences. IEEE, pp 1–5

  12. Friedrich G, Yeshurun Y (2002) Seeing people in the dark: face recognition in infrared images. In: Biologically Motivated Computer Vision. Springer, pp 348–359

  13. Ghiass RS, Arandjelović O, Bendada A, Maldague X (2014) Infrared face recognition: a comprehensive review of methodologies and databases. Pattern Recogn 47(9):2807–2824

    Article  Google Scholar 

  14. Gross R, Matthews I, Baker S (2006) Active appearance models with occlusion. Image Vis Comput 24(6):593–604

    Article  Google Scholar 

  15. Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Knyazev B, Kuharenko A, Junior JCSJ, Baró X, Demirel H, Allik J, Anbarjafari G (2018) Dominant and complementary emotion recognition from still images of faces. IEEE Access 6:26 391–26 403

    Article  Google Scholar 

  16. Haamer RE, Kulkarni K, Imanpour N, Haque MA, Avots E, Breisch M, Nasrollahi K, Escalera S, Ozcinar C, Baro X et al (2018) Changes in facial expression as biometric: a database and benchmarks of identification. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 621–628

  17. Hsieh C-C, Hsih M-H, Jiang M-K, Cheng Y-M, Liang E-H (2016) Effective semantic features for facial expressions recognition using svm. Multimed Tools Appl 75(11):6663–6682

    Article  Google Scholar 

  18. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456

  19. Jain A, Ross A, Prabhakar S (2004) An introduction to biometric recognition. IEEE Trans Circ Syst Video Technol 14(1):4–20

    Article  Google Scholar 

  20. Kulkarni K, Corneanu C, Ofodile I, Escalera S, Baró X, Hyniewska S, Allik J, Anbarjafari G (2018), Automatic recognition of facial displays of unfelt emotions. IEEE Transactions on Affective Computing

  21. Lin W-Y, Chen M-Y (2014) A novel framework for automatic 3d face recognition using quality assessment. Multimed Tools Appl 68(3):877–893

    Article  Google Scholar 

  22. Liu M, Wang R, Li S, Shan S, Huang Z, Chen X (2014) Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In: Proceedings of the 16th International Conference on Multimodal Interaction. ACM, pp 494–501

  23. Liu J, Liu W, Ma S, Lu C, Xiu X, Pathirage N, Li L, Chen G, Zeng W (2018) Face recognition based on manifold constrained joint sparse sensing with k-svd. Multimedia Tools and Applications 77(21):28863–28883

    Article  Google Scholar 

  24. Nikisins O, Nasrollahi K, Greitans M, Moeslund TB (2014) Rgb-dt based face recognition. In: 2014 22nd International Conference on Pattern Recognition (ICPR). IEEE, pp 1716–1721

  25. Nixon MS, Correia PL, Nasrollahi K, Moeslund TB, Hadid A, Tistarelli M (2015) On soft biometrics. Pattern Recogn Lett 68:218–230

    Article  Google Scholar 

  26. Odena A, Dumoulin V, Olah C (2016) Deconvolution and checkerboard artifacts. Distill 1(10):e3

    Article  Google Scholar 

  27. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition, vol 1

  28. Quan TM, Hilderbrand DG, Jeong W-K (2016) Fusionnet: A deep fully residual convolutional neural network for image segmentation in connectomics, arXiv:1612.05360

  29. Sajjadi MS, Scholkopf B, Hirsch M (2017) Enhancenet: Single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4491–4500

  30. Sarfraz MS, Stiefelhagen R (2015) Deep perceptual mapping for thermal to visible face recognition, arXiv:1507.02879

  31. Saxe AM, McClelland JL, Ganguli S (2013) Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, arXiv preprint arXiv:1312.6120

  32. Shi W, Caballero J, Huszár F, Totz J, Aitken A, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1874–1883

  33. Tompson J, Goroshin R, Jain A, LeCun Y, Bregler C (2015) Efficient object localization using convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 648–656

  34. van Laarhoven T (2017) L2 regularization versus batch and weight normalization, arXiv:1706.05350

  35. Wan J, Escalera S, Baro X, Escalante HJ, Guyon I, Madadi M, Allik J, Gorbova J, Anbarjafari G (2017) Results and analysis of chalearn lap multi-modal isolated and continuous gesture recognition, and real versus fake expressed emotions challenges. In: Chalearn lap, action, gesture, and emotion recognition workshop and competitions: Large scale multimodal gesture recognition and real versus fake expressed emotions, ICCV, vol 4, no 6

  36. Wilson AC, Roelofs R, Stern M, Srebro N, Recht B (2017) The marginal value of adaptive gradient methods in machine learning. In: Advances in Neural Information Processing Systems, pp 4151–4161

  37. Xu B, Wang N, Chen T, Li M (2015) Empirical evaluation of rectified activations in convolutional network, arXiv:1505.00853

  38. Zeiler MD, Krishnan D, Taylor GW, Fergus R (2010) Deconvolutional networks. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 2528–2535

  39. Zhang H, Patel VM, Riggan BS, Hu S (2017) Generative adversarial network-based synthesis of visible faces from polarimetrie thermal faces. In: 2017 IEEE International Joint Conference on Biometrics (IJCB). IEEE, pp 100–107

  40. Zhang T, Wiliem A, Yang S, Lovell B (2018) Tv-gan: Generative adversarial network based thermal to visible face recognition. In: 2018 International Conference on Biometrics (ICB). IEEE, pp 174–181

Download references

Acknowledgments

This work has been partially supported by IT Academy/StudyITin.ee, the Scientific and Technological Research Council of Turkey (TUBITAK) 1001 Project (116E097), by the Spanish project TIN2016-74946-P (MINECO/FEDER, UE) and CERCA Programme / Generalitat de Catalunya and the Estonian Centre of Excellence in IT (EXCITE) funded by the European Regional Development Fund. The authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of a Titan X Pascal GPU. This work is partially supported by ICREA under the ICREA Academia programme.

Author information

Authors and Affiliations

  1. iCV Research Group, Institute of Technology, University of Tartu, Tartu, 50411, Estonia

    Andre Litvin & Gholamreza Anbarjafari

  2. Visual Analysis of People Laboratory, Aalborg University, Aalborg, Denmark

    Kamal Nasrollahi & Thomas B. Moeslund

  3. University of Barcelona and Computer Vision Center, Barcelona, Spain

    Sergio Escalera

  4. School of Computer Science and Statistics, Trinity College Dublin, Dublin 2, Ireland

    Cagri Ozcinar

  5. Department of Electrical and Electronic Eng., Hasan Kalyoncu University, Gaziantep, Turkey

    Gholamreza Anbarjafari

  6. Institute of Digital Technologies, Loughborough University London, London, UK

    Gholamreza Anbarjafari

Authors
  1. Andre Litvin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kamal Nasrollahi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sergio Escalera
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cagri Ozcinar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thomas B. Moeslund
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gholamreza Anbarjafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gholamreza Anbarjafari.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Litvin, A., Nasrollahi, K., Escalera, S. et al. A novel deep network architecture for reconstructing RGB facial images from thermal for face recognition. Multimed Tools Appl 78, 25259–25271 (2019). https://doi.org/10.1007/s11042-019-7667-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7667-4

Keywords

Search

Navigation