Skip to main content

Advertisement

Springer Nature Link
Log in

Frontal face reconstruction based on detail identification, variable scale self-attention and flexible skip connection

  • Review
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Reconstruction of the frontal face from the profile is of great significance for face recognition in complex scenes. The existing mainstream methods of face reconstruction, such as FF-GAN, CAPG-GAN, TP-GAN, etc., have made good progresses on improving the generator network, but fewer considerations on the identification of face details and the extraction of spatial context features. To address the problem, we propose the frontal face reconstruction based on the detail discrimination, variable scale self attention, and flexible skip connection (FR-DVF): designing a group of discriminators for multi-scale detail region identification, a novel encoder-decoder generator structure with a variable scale type of self-attention module, which inserts a max-pooling layer into the pathways of the traditional module to reduce its feature-dimension and computing-cost, and a flexible type of the skip-connections to alleviate the stiff property of the traditional connections between the encoder and decoder layers. After adding detail discrimination, variable scale self attention module, and flexible skip connection structure, the rank-1 recognition rate (\(\%\)) of DVF-FR in the database of M2FPA increased by 2.94, 1.93 and 1.67\(\%\), respectively, as well as that occurred in FERET.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Akshay A, Marks Tim K, Jones Michael J, Tieu Kinh H, Rohith MV (2011) Fully automatic pose-invariant face recognition via 3d pose normalization. In: 2011 international conference on computer vision, pp 937–944

  2. Feng GC, Yuen PC (2000) Recognition of head-and-shoulder face image using virtual frontal-view image. IEEE Trans Syst Man Cybern Part A Syst Humans 30(6):871–882

    Article  Google Scholar 

  3. Guo Y, Juyong Z, Jianfei C, Boyi J, Jianmin J (2019) Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images. IEEE Trans Pattern Anal Mach Intell 41(6):1294–1307

    Article  Google Scholar 

  4. Liang S, Xiaoning S, Tao Z, Yuquan Z (2019) Histogram-based crc for 3d-aided pose-invariant face recognition. Sensors, 19(4)

  5. Hang Z, Jihao L, Ziwei L, Yu L, Xiaogang W (2020) Rotate-and-render: Unsupervised photorealistic face rotation from single-view images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), June

  6. Xi Y, Xiang Y, Kihyuk S, Xiaoming L, Manmohan C (2017) Towards large-pose face frontalization in the wild. In: Proceeding of international conference on computer vision, Venice, Italy, October

  7. Meina K, Shiguang S, Hong C, Xilin C (2014) Stacked progressive auto-encoders (spae) for face recognition across poses. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), June

  8. Forrester C, David B, Dilip K, Aaron S, Inbar M, Freeman William T (2017) Synthesizing normalized faces from facial identity features. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), July

  9. Xin Yu, Porikli F, Fernando B, Hartley R (2020) Hallucinating unaligned face images by multiscale transformative discriminative networks. Int J Comput Vision 128(2):500–526

    Article  Google Scholar 

  10. Junho Y, Heechul J, ByungIn Y, Changkyu C, Dusik P, Junmo K (2015) Rotating your face using multi-task deep neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), June

  11. Zhihong Zhang X, Chen BW, Guosheng H, Zuo W, Hancock ER (2019) Face frontalization using an appearance-flow-based convolutional neural network. IEEE Trans Image Process 28(5):2187–2199

    Article  MathSciNet  Google Scholar 

  12. Luan T, Xi Y, Xiaoming L (2017) Disentangled representation learning gan for pose-invariant face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1415–1424

  13. Rui H, Shu Z, Tianyu L, Ran H (2017) Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In: Proceedings of the IEEE international conference on computer vision (ICCV), Oct

  14. Yibo H, Xiang W, Bing Y, Ran H, Zhenan S (2018) Pose-guided photorealistic face rotation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June

  15. Dzmitry B, Kyunghyun C, Yoshua B (2016) Neural machine translation by jointly learning to align and translate

  16. Wei S, Tianfu W (2019) Learning spatial pyramid attentive pooling in image synthesis and image-to-image translation

  17. He Z, Kan M, Zhang J and Shan S (2020) Progressive attention generative adversarial network for facial attribute editing, Pa-gan

  18. Yu Y, Songyao J, Robinson Joseph P, Yun F (2020) Dual-attention gan for large-pose face frontalization. In: 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020), pp 249–256

  19. Yuhang L, Xuejin C, Feng W, Zheng Z (2019) Linestofacephoto: face photo generation from lines with conditional self-attention generative adversarial networks. MM ’19, pp 2323-2331, New York, NY, USA,. Association for Computing Machinery

  20. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR)

  21. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. Springer, Cham

    Google Scholar 

  22. Goodfellow IJ, Pouget-Abadie J, Mirza M, Bing X, Bengio Y (2014) Generative adversarial nets. MIT Press

  23. Jaderberg M, Simonyan K, Zisserman A et al (2015) Spatial transformer networks. Adv Neural Inf Process Syst 28:2017–2025

    Google Scholar 

  24. Xiaolong W, Ross G, Abhinav G, Kaiming H (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  25. Jie H, Li S, Gang S (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  26. Xiang L, Wenhai W, Xiaolin H, Jian Y (2019) Selective kernel networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 510–519

  27. Sanghyun W, Jongchan P, Joon-Young L, So KI (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  28. Jun-Yan Z, Taesung P, Phillip I, Efros Alexei A (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  29. Karen S, Andrew Z (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  30. Alex K, Ilya S, Hinton Geoffrey E (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inform Process Syst 25:1097–1105

    Google Scholar 

  31. Peipei L, Xiang W, Yibo H, Ran H, Zhenan S (2019) M2fpa: a multi-yaw multi-pitch high-quality dataset and benchmark for facial pose analysis. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10043–10051

  32. Jonathon PP, Harry W, Jeffery H, Rauss Patrick J (1998) The feret database and evaluation procedure for face-recognition algorithms. Image Vision Comput 16(5):295–306

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Nature Science Foundation of China under Grant 62061002.

Author information

Authors and Affiliations

  1. College of Electrical Engineering, Guangxi University, Nanning, 530004, China

    Haokun Luo, Shengcai Cen, Qichen Ding & Xueyun Chen

Authors
  1. Haokun Luo
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shengcai Cen
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Qichen Ding
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xueyun Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Xueyun Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, H., Cen, S., Ding, Q. et al. Frontal face reconstruction based on detail identification, variable scale self-attention and flexible skip connection. Neural Comput & Applic 34, 10561–10573 (2022). https://doi.org/10.1007/s00521-022-07124-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07124-5

Keywords