Skip to main content
SpringerLink
Log in

Challenges and Applications of Face Deepfake

  • Conference paper
  • First Online:
Frontiers of Computer Vision (IW-FCV 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1405))

Included in the following conference series:

Abstract

With the development of Generative deep learning algorithms in the last decade, it has become increasingly difficult to differentiate between what is real and what is fake. With the easily available “Deepfake” applications, even a person with less computing knowledge can also produce realistic Deepfake data. These fake data have many benefits while on the other hand, it can also be used for unethical and malicious purposes. Deepfake can be anything fake data generated by using deep learning methods. In this study, we focus on Deepfake with respect to face manipulation. We represent the currently used algorithms and datasets are represented for creating Deepfake. We also study the challenges and the real-world applications in which the benefits, as well as the drawbacks of using Deepfake, are being pointed out.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. 100k faces. https://generated.photos/. Accessed 14 Jan 2021

  2. 100k generated faces. https://github.com/NVlabs/stylegan. Accessed 14 Jan 2021

  3. AI enable deepfake. https://www.forbes.com/sites/bernardmarr/2019/07/22/the-best-and-scariest-examples-of-ai-enabled-deepfakes/?sh=86672662eaf1. Accessed 14 Jan 2021

  4. Amazon Mechanical Turk. https://www.mturk.com/. Accessed 14 Jan 2021

  5. BBC Obama. https://www.bbc.com/news/av/technology-40598465. Accessed 15 Jan 2021

  6. Computer-generated age progression photos. https://www.reddit.com/r/interestingasfuck/comments/kxf12x/the_accuracy_of_computergenerated_age_progression/. Accessed 15 Jan 2021

  7. Deep learning for detecting audiovisual fakes. https://sites.google.com/view/audiovisualfakes-icml2019/. Accessed 14 Jan 2021

  8. Deepfake Salvador the Verge. https://www.theverge.com/2019/5/10/18540953/salvador-dali-lives-deepfake-museum. Accessed 14 Jan 2021

  9. Deepfake video of Mark Zuckerberg. https://finance.yahoo.com/news/deepfake-video-mark-zuckerberg-goes-163128674.html?guccounter=1. Accessed 15 Jan 2021

  10. Deepfaketimit. https://www.idiap.ch/dataset/deepfaketimit. Accessed 14 Jan 2021

  11. Deepnude. https://www.vox.com/2019/6/27/18761639/ai-deepfake-deepnude-app-nude-women-porn. Accessed 15 Jan 2021

  12. Dimentions. https://app.dimensions.ai/. Accessed 15 Jan 2021

  13. Facebook AI deepFake detection challenge dataset. https://ai.facebook.com/datasets/dfdc/. Accessed 14 Jan 2021

  14. Faceswap. https://github.com/deepfakes/faceswap. Accessed 14 Jan 2021

  15. Faceswap. https://github.com/MarekKowalski/FaceSwap/. Accessed 14 Jan 2021

  16. Fakeapp. https://fakeapp.softonic.com/. Accessed 14 Jan 2021

  17. FFHQ dataset. https://github.com/NVlabs/ffhq-dataset. Accessed 14 Jan 2021

  18. FFmpeg. https://ffmpeg.org/. Accessed 14 Jan 2021

  19. Forbes digital doubles. https://www.forbes.com/sites/katiebaron/2019/07/29/digital-doubles-the-deepfake-tech-nourishing-new-wave-retail/?sh=4e656bac4cc7/. Accessed 14 Jan 2021

  20. Making Robert de Niro in “the Irishman”. https://www.businessinsider.com/deepfake-netflix-correcting-the-irishman-de-ageing-tech-2020-1. Accessed 14 Jan 2021

  21. NIST media forensics challenge 2018. https://www.nist.gov/itl/iad/mig/media-forensics-challenge-2018. Accessed 14 Jan 2021

  22. Retailwire. https://retailwire.com/discussion/can-deepfake-technology-reduce-retail-returns-without-rattling-reality/. Accessed 14 Jan 2021

  23. Reuters David Beckham’s ‘deep fake’ malaria awareness video. https://mobile.reuters.com/video/watch/david-beckhams-deep-fake-malaria-awarene-id536254167?chan=c1tal5kh. Accessed 15 Jan 2021

  24. The Verge Barack Obama. https://www.theverge.com/tldr/2018/4/17/17247334/ai-fake-news-video-barack-obama-jordan-peele-buzzfeed. Accessed 15 Jan 2021

  25. Vice. https://www.vice.com/en/article/gydydm/gal-gadot-fake-ai-porn. Accessed 15 Jan 2021

  26. Vidtimit. https://conradsanderson.id.au/vidtimit/. Accessed 14 Jan 2021

  27. Workshop on media forensics. https://sites.google.com/view/mediaforensics2019. Accessed 14 Jan 2021

  28. Barratt, S., Sharma, R.: A note on the inception score. arXiv preprint arXiv:1801.01973 (2018)

  29. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2018)

    Google Scholar 

  30. Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 539–546. IEEE (2005)

    Google Scholar 

  31. Dolhansky, B., et al.: The deepfake detection challenge dataset. arXiv preprint arXiv:2006.07397 (2020)

  32. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)

    Google Scholar 

  33. Georghiades, A.S., Belhumeur, P.N., Kriegman, D.J.: From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 643–660 (2001)

    Article  Google Scholar 

  34. Goodfellow, I., et al.: Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27, 2672–2680 (2014)

    Google Scholar 

  35. He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: AttGAN: facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28(11), 5464–5478 (2019)

    Article  MathSciNet  Google Scholar 

  36. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501–1510 (2017)

    Google Scholar 

  37. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462–2470 (2017)

    Google Scholar 

  38. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

    Google Scholar 

  39. Ito, K., Xiong, K.: Gaussian filters for nonlinear filtering problems. IEEE Trans. Autom. Control 45(5), 910–927 (2000)

    Article  MathSciNet  Google Scholar 

  40. Jiang, L., Li, R., Wu, W., Qian, C., Loy, C.C.: Deeperforensics-1.0: a large-scale dataset for real-world face forgery detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2886–2895. IEEE (2020)

    Google Scholar 

  41. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017)

  42. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)

    Google Scholar 

  43. Kim, H., et al.: Deep video portraits. ACM Trans. Graph. (TOG) 37(4), 1–14 (2018)

    Google Scholar 

  44. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  45. Kingma, D.P., Mohamed, S., Jimenez Rezende, D., Welling, M.: Semi-supervised learning with deep generative models. Adv. Neural Inf. Process. Syst. 27, 3581–3589 (2014)

    Google Scholar 

  46. Korshunov, P., Marcel, S.: Deepfakes: a new threat to face recognition? Assessment and detection. arXiv preprint arXiv:1812.08685 (2018)

  47. Korshunova, I., Shi, W., Dambre, J., Theis, L.: Fast face-swap using convolutional neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3677–3685 (2017)

    Google Scholar 

  48. Lample, G., Zeghidour, N., Usunier, N., Bordes, A., Denoyer, L., Ranzato, M.: Fader networks: manipulating images by sliding attributes. In: Advances in Neural Information Processing Systems, pp. 5967–5976 (2017)

    Google Scholar 

  49. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H., Hawk, S.T., Van Knippenberg, A.: Presentation and validation of the Radboud faces database. Cogn. Emot. 24(8), 1377–1388 (2010)

    Article  Google Scholar 

  50. Lee, C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: towards diverse and interactive facial image manipulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5549–5558 (2020)

    Google Scholar 

  51. Li, C., Wand, M.: Combining Markov random fields and convolutional neural networks for image synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2479–2486 (2016)

    Google Scholar 

  52. Li, Y., Chang, M.C., Lyu, S.: In Ictu Oculi: exposing AI generated fake face videos by detecting eye blinking. arXiv preprint arXiv:1806.02877 (2018)

  53. Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for deepfake forensics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3207–3216 (2020)

    Google Scholar 

  54. Liu, M., et al.: StGAN: a unified selective transfer network for arbitrary image attribute editing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3673–3682 (2019)

    Google Scholar 

  55. Liu, Y., Fan, H., Ni, F., Xiang, J.: ClsGAN: selective attribute editing model based on classification adversarial network. Neural Netw. 133, 220–228 (2017)

    Article  Google Scholar 

  56. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

    Google Scholar 

  57. Mathiasen, A., Hvilshøj, F.: Fast fr\(\backslash \)’echet inception distance. arXiv preprint arXiv:2009.14075 (2020)

  58. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)

  59. Natsume, R., Yatagawa, T., Morishima, S.: FsNet: an identity-aware generative model for image-based face swapping. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 117–132. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-20876-9_8

    Chapter  Google Scholar 

  60. Natsume, R., Yatagawa, T., Morishima, S.: RsGAN: face swapping and editing using face and hair representation in latent spaces. arXiv preprint arXiv:1804.03447 (2018)

  61. Neves, J.C., Tolosana, R., Vera-Rodriguez, R., Lopes, V., Proença, H., Fierrez, J.: GANPrintr: improved fakes and evaluation of the state of the art in face manipulation detection. arXiv preprint arXiv:1911.05351 (2019)

  62. Nirkin, Y., Masi, I., Tuan, A.T., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 98–105. IEEE (2018)

    Google Scholar 

  63. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301. IEEE (2009)

    Google Scholar 

  64. Perarnau, G., Van De Weijer, J., Raducanu, B., Álvarez, J.M.: Invertible conditional GANs for image editing. arXiv preprint arXiv:1611.06355 (2016)

  65. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH 2003 Papers, pp. 313–318 (2003)

    Google Scholar 

  66. Prajwal, K., Mukhopadhyay, R., Namboodiri, V.P., Jawahar, C.: A lip sync expert is all you need for speech to lip generation in the wild. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 484–492 (2020)

    Google Scholar 

  67. Reinhard, E., Adhikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)

    Article  Google Scholar 

  68. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: Faceforensics: a large-scale video dataset for forgery detection in human faces. arXiv preprint arXiv:1803.09179 (2018)

  69. Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: Faceforensics++: learning to detect manipulated facial images. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1–11 (2019)

    Google Scholar 

  70. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  71. Solon, O.: Facial recognition’s ‘dirty little secret’: millions of online photos scraped without consent. NBC News (2019)

    Google Scholar 

  72. Stehouwer, J., Dang, H., Liu, F., Liu, X., Jain, A.: On the detection of digital face manipulation. arXiv preprint arXiv:1910.01717 (2019)

  73. Tao, R., Li, Z., Tao, R., Li, B.: Resattr-GAN: unpaired deep residual attributes learning for multi-domain face image translation. IEEE Access 7, 132594–132608 (2019)

    Article  Google Scholar 

  74. Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)

    Article  Google Scholar 

  75. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395 (2016)

    Google Scholar 

  76. Tuan Tran, A., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5163–5172 (2017)

    Google Scholar 

  77. Wang, X., Yu, K., Dong, C., Loy, C.C.: Recovering realistic texture in image super-resolution by deep spatial feature transform. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  78. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  79. Welch, G., Bishop, G., et al.: An introduction to the Kalman filter (1995)

    Google Scholar 

  80. Wu, P.W., Lin, Y.J., Chang, C.H., Chang, E.Y., Liao, S.W.: RelGAN: multi-domain image-to-image translation via relative attributes. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5914–5922 (2019)

    Google Scholar 

  81. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  82. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

  83. Zhu, P., Abdal, R., Qin, Y., Wonka, P.: Sean: image synthesis with semantic region-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5104–5113 (2020)

    Google Scholar 

  84. Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 146–155 (2016)

    Google Scholar 

  85. Zi, B., Chang, M., Chen, J., Ma, X., Jiang, Y.G.: Wilddeepfake: a challenging real-world dataset for deepfake detection. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2382–2390 (2020)

    Google Scholar 

Download references

Acknowledgment

This study was supported by the BK21 FOUR project (AI-driven Convergence Software Education Research Program) funded by the Ministry of Education, School of Computer Science and Engineering, Kyungpook National University, Korea (4199990214394).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Kyungpook National University, Daegu, Republic of Korea

    Lamyanba Laishram, Md. Maklachur Rahman & Soon Ki Jung

Authors
  1. Lamyanba Laishram
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Md. Maklachur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Soon Ki Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Soon Ki Jung .

Editor information

Editors and Affiliations

  1. Chonnam National University, Gwangju, Korea (Republic of)

    Hieyong Jeong

  2. Aoyama Gakuin University, Kanagawa, Japan

    Kazuhiko Sumi

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Laishram, L., Rahman, M.M., Jung, S.K. (2021). Challenges and Applications of Face Deepfake. In: Jeong, H., Sumi, K. (eds) Frontiers of Computer Vision. IW-FCV 2021. Communications in Computer and Information Science, vol 1405. Springer, Cham. https://doi.org/10.1007/978-3-030-81638-4_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-81638-4_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-81637-7

  • Online ISBN: 978-3-030-81638-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation