Skip to main content
Springer Nature Link
Log in

ATFS: A deep learning framework for angle transformation and face swapping of face de-identification

  • 1230: Sentient Multimedia Systems and Visual Intelligence
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes an effective framework for improving angle transformation and face replacement. This ATFS framework aims to solve the image distortion caused by angled face swapping in face de-identification. In the ATFS framework, TransNet and SWGAN were proposed based on generative adversarial networks. When TransNet uses neural networks to reconstruct images, it combines the predicted facial points. As a result, Transnet can locate the target’s facial features and generate multi-angle transformed images to achieve angle transformation. The SWGAN applies a complicated neural network that combines residual operations and self-attention modules to extract the input image features and replace the face region of the input image with the target image to achieve face de-identification. Both TransNet and SWGAN adopt discriminators and various loss functions to train the neural network and accurately accelerate the training process. The experimental results show that the proposed method can maintain high-quality images and avoid image distortion during face swapping for image angle transformation compared to previously proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Amelio, A., Bonifazi, G., Cauteruccio, F., Corradini, E., Marchetti, M., Ursino, D., Virgili, L.: Representation and compression of residual neural networks through a multilayer network based approach. Expert Systems with Applications 215, 119391 (2023). doi: https://doi.org/10.1016/j.eswa.2022.119391.https://www.sciencedirect.com/science/article/pii/S0957417422024095

  2. Bakkouri, I., Afdel, K.: Computer-aided diagnosis (cad) system based on multi-layer feature fusion network for skin lesion recognition in dermoscopy images. Multimedia Tools and Applications 79(29–30), 20483–20518 (2020)

    Article  Google Scholar 

  3. Bakkouri, I., Afdel, K.: Mlca2f: Multi-level context attentional feature fusion for covid-19 lesion segmentation from ct scans. Signal, Image and Video Processing pp. 1–8 (2022)

    Google Scholar 

  4. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030 (2017)

  5. Chen, D., Chang, Y., Yan, R., Yang, J.: Protecting personal identification in video. In: Protecting Privacy in Video Surveillance, pp. 115–128. Springer (2009)

  6. Chen, D., Chen, Q., Wu, J., Yu, X., Jia, T.: Face swapping: realistic image synthesis based on facial landmarks alignment. Mathematical Problems in Engineering 2019 (2019)

  7. Cho, D., Lee, J.H., Suh, I.H.: Cleanir: Controllable attribute-preserving natural identity remover. Applied Sciences 10(3), 1120 (2020)

    Article  Google Scholar 

  8. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Transactions on pattern analysis and machine intelligence 23(6), 681–685 (2001)

    Article  Google Scholar 

  9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)

  10. Iglovikov, V., Shvets, A.: Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation. arXiv preprint arXiv:1801.05746 (2018)

  11. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1867–1874 (2014)

  12. Lin, C.H., Wang, Z.H., Jong, G.J.: A de-identification face recognition using extracted thermal features based on deep learning. IEEE Sensors Journal 20(16), 9510–9517 (2020)

    Google Scholar 

  13. Meden, B., Mallı, R.C., Fabijan, S., Ekenel, H.K., Štruc, V., Peer, P.: Face deidentification with generative deep neural networks. IET Signal Processing 11(9), 1046–1054 (2017)

    Article  Google Scholar 

  14. Naruniec, J., Helminger, L., Schroers, C., Weber, R.M.: High-resolution neural face swapping for visual effects. In: Computer Graphics Forum, vol. 39, pp. 173–184. Wiley Online Library (2020)

  15. Nirkin, Y., Keller, Y., Hassner, T.: Fsgan: Subject agnostic face swapping and reenactment. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 7184–7193 (2019)

  16. Nirkin, Y., Keller, Y., Hassner, T.: Fsganv 2: Improved subject agnostic face swapping and reenactment. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(1), 560–575 (2022)

    Article  Google Scholar 

  17. Nirkin, Y., Masi, I., Tuan, A.T., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 98–105. IEEE (2018)

  18. Perov, I., Gao, D., Chervoniy, N., Liu, K., Marangonda, S., Umé, C., Dpfks, M., Facenheim, C.S., RP, L., Jiang, J., et al.: Deepfacelab: Integrated, flexible and extensible face-swapping framework. arXiv preprint arXiv:2005.05535 (2020)

  19. Ribaric, S., Pavesic, N.: An overview of face de-identification in still images and videos. In: 2015 11th IEEE International conference and workshops on automatic face and gesture recognition (FG), vol. 4, pp. 1–6. IEEE (2015)

  20. Rowley, H.A., Baluja, S., Kanade, T.: Neural network-based face detection. IEEE Transactions on pattern analysis and machine intelligence 20(1), 23–38 (1998)

    Article  Google Scholar 

  21. Samarzija, B., Ribaric, S.: An approach to the de-identification of faces in different poses. In: 2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1246–1251. IEEE (2014)

  22. Shu, C., Wu, H., Zhou, H., Liu, J., Hong, Z., Ding, C., Han, J., Liu, J., Ding, E., Wang, J.: Few-shot head swapping in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10789–10798 (2022)

  23. Souček, T., Moravec, J., Lokoč, J.: Transnet: A deep network for fast detection of common shot transitions. arXiv preprint arXiv:1906.03363 (2019)

  24. Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5693–5703 (2019)

  25. Tuan Tran, A., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3d morphable models with a very deep neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5163–5172 (2017)

  26. Umeyama, S.: Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence 13(04), 376–380 (1991)

    Article  Google Scholar 

  27. Wu, J., Huang, Z., Acharya, D., Li, W., Thoma, J., Paudel, D.P., Gool, L.V.: Sliced wasserstein generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3713–3722 (2019)

  28. Wu, Y., Yang, F., Xu, Y., Ling, H.: Privacy-protective-gan for privacy preserving face de-identification. Journal of Computer Science and Technology 34(1), 47–60 (2019)

    Article  Google Scholar 

  29. Xu, Y., Deng, B., Wang, J., Jing, Y., Pan, J., He, S.: High-resolution face swapping via latent semantics disentanglement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7642–7651 (2022)

Download references

Funding

This research was funded by Ministry of Science and Technology grant number MOST 109-2221-E-005-057-MY2 and .

Author information

Authors and Affiliations

  1. Department of Management Information Systems, National Chung Hsing University, Taichung, 402, Taiwan

    Chwei-Shyong Tsai, Wei-Ting Chen & Josh Jia-Ching Ying

  2. Language Center, National Chin-Yi University of Technology, Taichung, 411, Taiwan

    Hsien-Chu Wu

  3. Department of Computer Science and Information Engineering, National Taichung University of Science and Technology, Taichung, 404, Taiwan

    Hsien-Chu Wu

Authors
  1. Chwei-Shyong Tsai
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Hsien-Chu Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Wei-Ting Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Josh Jia-Ching Ying
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Josh Jia-Ching Ying.

Ethics declarations

Conflicts of interests/Competing interests

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tsai, CS., Wu, HC., Chen, WT. et al. ATFS: A deep learning framework for angle transformation and face swapping of face de-identification. Multimed Tools Appl 83, 36797–36822 (2024). https://doi.org/10.1007/s11042-023-16123-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16123-0

Keywords