Skip to main content
SpringerLink
Log in

FRAN: feature-filtered residual attention network for realistic face sketch-to-photo transformation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Face sketch-to-photo transformation aims at generates face photo images from sketched face images. Although transformations have progressed significantly with the development of deep learning techniques in recent years, generating face photos with realistic photo styles and rich facial details is still challenging. In this paper, a new realistic face sketch-to-photo transformation method is proposed based on the feature-filtered residual attention network (FRAN), which is able to propagate more precise feature information in the deep network. Specifically, a feature-filtered residual module is constructed by filtering feature maps in the residual block to filtrate short-term feature information. In addition, a decoder-guided attention module is designed to integrate and filtrate the long-term feature information. Moreover, to synthesize face photo images with more facial details, a Sobel operator-based detail loss is proposed to constrain the network training. The experimental results on the public datasets demonstrate that FRAN generates more realistic face photo images than state-of-the-art approaches in terms of visual perception and quality evaluation. Furthermore, the face photo images generated by FRAN obtain higher face recognition accuracy than those created by the compared methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Yang S, Wang Z, Liu J, Guo Z (2021) Controllable sketch-to-image translation for robust face synthesis. IEEE Trans Image Process 30:8797–8810

    Article  Google Scholar 

  2. Wan W, Yang Y, Lee HJ (2021) Generative adversarial learning for detail-preserving face sketch synthesis. Neurocomputing 438:107–121

    Article  Google Scholar 

  3. Zhang Y, Yu L, Sun B, He J (2022) ENG-face: cross-domain heterogeneous face synthesis with enhanced asymmetric CycleGAN. Appl Intell 52:15295–15307. https://doi.org/10.1007/s10489-022-03302-z

    Article  Google Scholar 

  4. Yu J, Xu X, Gao F, Shi S, Wang M, Tao D, Huang Q (2021) Toward realistic face photo–sketch synthesis via composition-aided GANs. IEEE Trans Cybern 51(9):4350–4362

    Article  Google Scholar 

  5. Tang X, Wang X (2002) Face photo recognition using sketch. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp 257-260

  6. Song Y, Bao L, Yang Q, Yang MH (2014) Real-time exemplar-based face sketch synthesis. In: European Conference on Computer Vision (ECCV), pp 800–813

  7. Wang N, Gao X, Li J (2018) Random sampling for fast face sketch synthesis. Pattern Recogn 76:215–227

    Article  Google Scholar 

  8. Jiao L, Zhang S, Li L, Liu F, Ma W (2018) A modified convolutional neural network for face sketch synthesis. Pattern Recogn 76:439–446

    Article  Google Scholar 

  9. Sheng B, Li P, Gao C, Ma K (2018) Deep neural representation guided face sketch synthesis. IEEE Trans Vis Comput Graph 25(12):3216–3230

    Article  Google Scholar 

  10. Yan L, Zheng W, Gou C, Wang F (2021) IsGAN: identity-sensitive generative adversarial network for face photo-sketch synthesis. Pattern Recogn 119:108077

    Article  Google Scholar 

  11. Zhang C, Liu D, Peng C, Wang N, Gao X (2022) Edge aware domain transformation for face sketch synthesis. IEEE Trans Inf Forensic Secur 17:2761–2770

    Article  Google Scholar 

  12. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  Google Scholar 

  13. Gao X, Wang N, Tao D, Li X (2012) Face sketch-photo synthesis and retrieval using sparse representation. IEEE Trans Circ Syst Video Technol 22(8):1213–1226

    Article  Google Scholar 

  14. Li J, Yu X, Peng C, Wang N (2017) Adaptive representation-based face sketch-photo synthesis. Neurocomputing 269:152–159

    Article  Google Scholar 

  15. Peng C, Gao X, Wang N, Li J (2017) Superpixel-based face sketch–photo synthesis. IEEE Trans Circ Syst Video Technol 27(2):288–299

    Article  Google Scholar 

  16. Zhang M, Wang R, Gao X, Li J, Tao D (2019) Dual-transfer face sketch–photo synthesis. IEEE Trans Image Process 28(2):642–657

    Article  MathSciNet  MATH  Google Scholar 

  17. Lin Y, Fu K, Ling S, Wang J, Cheng P (2022) Toward identity preserving face synthesis between sketches and photos using deep feature injection. IEEE Trans Ind Inf 18(1):327–336

    Article  Google Scholar 

  18. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, WardeFarley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Communications of the ACM 63(11):139–144

  19. Yadav NK, Singh SK, Dubey SR (2022) CSA-GAN: cyclic synthesized attention guided generative adversarial network for face synthesis. Appl Intell 52:12704–12723. https://doi.org/10.1007/s10489-021-03064-0

    Article  Google Scholar 

  20. Kazemi H, Iranmanesh M, Dabouei A, Soleymani S, Nasrabadi N M (2018) Facial attributes guided deep sketch-to-photo synthesis. In Proceedings of IEEE Winter Conference on Application of Computer Vision Workshops, pp 1–8

  21. Wang L, Sindagi V A, Patel V M (2018) High-quality facial photo-sketch synthesis using multi-adversarial networks. In Proceedings of 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp 83–90

  22. Chao W, Chang L, Wang X, Cheng J, Deng X, Duan F (2019) High-fidelity face sketch-to-photo synthesis using generative adversarial network. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp 4699-4703

  23. Lei Y, Du W, Hu Q (2020) Face sketch-to-photo transformation with multi-scale self-attention GAN. Neurocomputing 396:12–23

    Article  Google Scholar 

  24. Duan S, Chen Z, Wu QMJ, Cai L, Lu D (2021) Multi-scale gradients self-attention residual learning for face photo-sketch transformation. IEEE Trans Inf Forensic Secur 16:1218–1230

    Article  Google Scholar 

  25. Li L, Tang J, Shao Z, Tan X, Ma L (2022) Sketch-to-photo face generation based on semantic consistency preserving and similar connected component refinement. Vis Comput 38(11):3577–3594

  26. Jing Y, Yang Y, Feng Z, Ye J, Yu Y, Song M (2019) Neural style transfer: a review. IEEE Trans Vis Comput Graph 26(11):3365–3385

    Article  Google Scholar 

  27. Jam J, Kendrick C, Walker K, Drouard V, Hsu JG, Yap MH (2021) A comprehensive review of past and present image inpainting methods. Comput Vis Image Underst 203:103147

    Article  Google Scholar 

  28. Wan W, Lee H J (2019) Generative adversarial multi-task learning for face sketch synthesis and recognition. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp 4065-4069

  29. Isola P, Zhu J Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1125–1134

  30. Wang T, Liu M, Zhu J, Tao A, Kautz J, Catanzaro B (2018) High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798–8807

  31. Zhu J Y, Park T, Isola P, Efros A A (2017) Unpaired image-to image translation using cycle-consistent adversarial networks. In: International Conference on Computer Vision, pp 2223–2232

  32. Yi Z, Zhang H, Tan P, Gong M (2017) DualGAN: unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2849–2857

  33. Kim T, Cha M, Kim H, Lee J K, Kim J (2017) Learning to discover cross-domain relations with generative adversarial networks. In: Proceedings of International Conference on Machine Learning, pp 1857–1865

  34. Babu KK, Dubey SR (2021) CSGAN: cyclic-synthesized generative adversarial networks for image-to-image transformation. Expert Syst Appl 169:114431

    Article  Google Scholar 

  35. Ji Y, Zhang H, Wu QMJ (2018) Salient object detection via multi-scale attention CNN. Neurocomputing 322:130–140

    Article  Google Scholar 

  36. Chang W Y, Tsai M Y, Lo S C (2021) ResSaNet: a hybrid backbone of residual block and self-attention module for masked face recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1468-1476

  37. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3156-3164

  38. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7794-7803

  39. Woo S, Park J, Lee J Y, Kweon I S. (2018) CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 3-19

  40. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7132-7141

  41. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3146–3154

  42. Lu E, Hu X (2022) Image super-resolution via channel attention and spatial attention. Appl Intell 52(2):2260–2268

    Article  MathSciNet  Google Scholar 

  43. Vairalkar MK, Nimbhorkar SU (2012) Edge detection of images using Sobel operator. Int J Emerg Technol Adv Eng 2(1):291–293

    Google Scholar 

  44. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of European Conference on Computer Vision, 694–711

  45. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

    Google Scholar 

  46. Wang L, Sindagi V, Patel V (2018) High-quality facial photo-sketch synthesis using multi-adversarial networks. In: Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition, pp 83-90

  47. Kim J, Kim M, Kang H, Lee K (2020) U-GAT-IT: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In: Proceedings of International Conference on Learning Representations, pp 1-11

  48. Tang H, Liu H, Xu D, Torr PHS, Sebe N (2021) AttentionGAN: unpaired image-to-image translation using attention-guided generative adversarial networks. IEEE Trans Neural Netw Learn Syst:1–16. https://doi.org/10.1109/TNNLS.2021.3105725

  49. Martinez AM, Benavente R (1998) The AR face database. CVC Technical Report #24

  50. Messer K, Matas J, Kittler J, Luettin J, Maitre G (1999) XM2VTSDB: The extended of M2VTS database. In: Proceedings of International Conference on Audio- and Video-Based Person Authentication, pp 72–77

  51. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

    Google Scholar 

  52. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  53. Zhang L, Zhang L, Mou X, Zhang D (2011) FSIM: a feature similarity index for image quality assessment. IEEE Trans Image Process 20(8):2378–2386

    Article  MathSciNet  MATH  Google Scholar 

  54. Wan W, Lee HJ (2019) A joint training model for face sketch synthesis. Appl Sci 9:1731

    Article  Google Scholar 

  55. Cao Q, Shen L, Xie W, Parkhi O M, Zisserman A (2018) VGGFace2: a dataset for recognising faces across pose and age. In: Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition, pp 67-74

Download references

Acknowledgments

This study has been supported in part by the National Natural Science Foundation of China (62261025, 62262023, 62072218, 61862030), by the Natural Science Foundation of Jiangxi Province (20192ACB20002, 20192ACBL21008), by the Project of the Education Department of Jiangxi Province (GJJ200541), and by the Postdoctoral Research Projects of Jiangxi Province (2020KY44).

Author information

Authors and Affiliations

  1. School of Software and Internet of Things Engineering, Jiangxi University of Finance and Economics, Nanchang, China

    Weiguo Wan

  2. School of Computer Science and Technology, Tiangong University, Tianjin, China

    Yong Yang

  3. School of Software, Tiangong University, Tianjin, China

    Shuying Huang

  4. School of Mathematics and Computer Science, Jiangxi Science and Technology Normal University, Nanchang, China

    Lixin Gan

Authors
  1. Weiguo Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuying Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lixin Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong Yang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wan, W., Yang, Y., Huang, S. et al. FRAN: feature-filtered residual attention network for realistic face sketch-to-photo transformation. Appl Intell 53, 15946–15956 (2023). https://doi.org/10.1007/s10489-022-04352-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04352-z

Keywords

Search

Navigation