Skip to main content

Exposing deepfake using fusion of deep-learned and hand-crafted features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the milliards of images and videos, visual content has become a critical source of information. The spread of misinformation has become a significant problem through the availability of editing tools, requiring robust manipulation detection methods. Some of the manipulations, such as copy-move and splicing, are easy to detect, while other advanced facial manipulations, such as DeepFake are hard to detect. Facial manipulations can change human expressions by creating highly realistic faces. In this paper, we propose an efficient method to expose DeepFake in digital videos. A fusion of hand-crafted and deep-learned features is utilized to improve detection performance. The image quality measure (FM) is used besides the similarity measure of face and body skin color to generate the hand-crafted features. The experimental results show the efficiency of the proposed method for exposing DeepFake. We conducted the experiments on the three commonly and publicly available datasets Celeb-df, DFDC, and Faceforensics++.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Algorithm 1
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

The datasets used and/or analyzed during the current study are available in the Celeb-DF, DFDC, and FaceForensics++ repositories respectively,

http://www.cs.albany.edu/lsw/celeb-deepfakeforensics.html

https://www.kaggle.com/c/deepfake-detection-challenge

https://github.com/ondyari/FaceForensics

Notes

  1. Masayuki Tanaka (2021). Face Parts Detection https://www.mathworks.com/matlabcentral/fileexchange/36855-face-parts-detection, MATLAB Central File Exchange. Retrieved September 4, 2021.

  2. http://www.cs.albany.edu/lsw/celeb-deepfakeforensics.html

  3. https://www.kaggle.com/c/deepfake-detection-challenge

  4. https://github.com/ondyari/FaceForensics

References

  1. Afchar, D, Nozick, V, Yamagishi, J, Echizen, I (2018) Mesonet: a compact facial video forgery detection network. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS), p. 1–7. IEEE

  2. Bakas J, Naskar R, Dixit R (2019) Detection and localization of inter-frame video forgeries based on inconsistency in correlation distribution between haralick coded frames. Multimed Tools Appl 78(4):4905–4935. https://doi.org/10.1007/s11042-018-6570-8

    Article  Google Scholar 

  3. Bayar, B, Stamm, MC(2016) A deep learning approach to universal image manipulation detection using a new convolutional layer. In: Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security, p. 5–10. ACM

  4. Boylan, JF(2018) The New York Times. Will DeepFake Technology Destroy Democracy? https://www.nytimes.com/2018/10/17/opinion/deep-fake-technology-democracy.html

  5. Chatfield, K, Simonyan, K, Vedaldi, A, Zisserman, A (2014) Return of the devil in the details: Delving deep into convolutional nets. In: British Machine Vision Conference

  6. Cozzolino, D, Poggi, G, Verdoliva, L(2017) Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection. In: Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security, p. 159–164. ACM

  7. De K, Masilamani V (2013) Image sharpness measure for blurred images in frequency domain. Procedia Eng 64:149–158

    Article  Google Scholar 

  8. Detection of inter-frame forgeries in digital videos (2018) K., S., Mehtre, B.M. Forensic Sci Int 289:186–206. https://doi.org/10.1016/j.forsciint.2018.04.056

    Article  Google Scholar 

  9. Dolhansky, B, Howes, R, Pflaum, B, Baram, N, Ferrer, CC (2019) The deepfake detection challenge (dfdc) preview dataset. arXiv:1910.08854

  10. Elaskily MA, Elnemr HA, Dessouky MM, Faragallah OS (2019) Two stages object recognition based copy-move forgery detection algorithm. Multimed Tools Appl. 78(11):15353–15373. https://doi.org/10.1007/s11042-018-6891-7

    Article  Google Scholar 

  11. Fadl, S, Han, Q, Qiong, L (2020) Exposing video inter-frame forgery via histogram of oriented gradients and motion energy image. Multidimens. Syst. Signal Process, 1–20

  12. Fadl SM, Semary NA (2017) Robust copy-move forgery revealing in digital images using polar coordinate system. Neurocomputing 265:57–65. https://doi.org/10.1016/j.neucom.2016.11.091

    Article  Google Scholar 

  13. Fridrich J, Kodovsky J (2012) Rich models for steganalysis of digital images. IEEE Trans. Inf. Forensics Secur 7(3):868–882. https://doi.org/10.1109/TIFS.2012.2190402

    Article  Google Scholar 

  14. Fung, S, Lu, X, Zhang, C, Li, C-T(2021) Deepfakeucl: Deepfake detection via unsupervised contrastive learning. arXiv:2104.11507

  15. Fung, S, Lu, X, Zhang, C, Li, C-T(2021) Deepfakeucl: Deepfake detection via unsupervised contrastive learning. In: 2021 International Joint Conference on Neural Networks (IJCNN), p. 1–8. IEEE

  16. Gatys, LA, Ecker, AS, Bethge, M (2015) A neural algorithm of artistic style. arXiv:1508.06576

  17. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Commun ACM 63(11):139–144

    Article  MathSciNet  Google Scholar 

  18. Jain AK, Flynn P, Ross AA (2007) Handbook of Biometrics. Springer

    Google Scholar 

  19. Juefei-Xu, F, Wang, R, Huang, Y, Guo, Q, Ma, L, Liu, Y (2021) Countering malicious deepfakes: Survey, battleground, and horizon. arXiv:2103.00218

  20. Khalid, H, Woo, SS(2020) Oc-fakedect: Classifying deepfakes using one-class variational autoencoder. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, p. 656–657

  21. Korshunov, P, Marcel, S(2018) Deepfakes: a new threat to face recognition? Assessment and detection. arXiv:1812.08685

  22. Korshunova, I, Shi, W, Dambre, J, Theis, L(2017) Fast face-swap using convolutional neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, p. 3677–3685

  23. Kumar, P, Vatsa, M, Singh, R (2020) Detecting face2face facial reenactment in videos. In: The IEEE Winter Conference on Applications of Computer Vision (WACV)

  24. Laws, KI(1980) Textured image segmentation. Technical report, University of Southern California Los Angeles Image Processing INST

  25. Li, Y, Lyu, S(2018) Exposing deepfake videos by detecting face warping artifacts. arXiv:1811.00656

  26. Li, Y, Yang, X, Sun, P, Qi, H, Lyu, S(2020) Celeb-df: A large-scale challenging dataset for deepfake forensics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3207–3216

  27. Li H, Li B, Tan S, Huang J (2020) Identification of deep network generated images using disparities in color components. Signal Process 174:107616

    Article  Google Scholar 

  28. Matern, F, Riess, C, Stamminger, M(2019) Exploiting visual artifacts to expose deepfakes and face manipulations. In: 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), p. 83–92. https://doi.org/10.1109/WACVW.2019.00020

  29. McCloskey, S, Albright, M(2019) Detecting gan-generated imagery using saturation cues. In: 2019 IEEE International Conference on Image Processing (ICIP), p. 4584–4588. IEEE

  30. Megahed, A., Fadl, S.M., Han, Q., Li, Q(2017) Handwriting forgery detection based on ink colour features. In: 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), pp. 141–144. IEEE

  31. Megahed, A, Han, Q(2020) Face2face manipulation detection based on histogram of oriented gradients. In: 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), p. 1260–1267. https://doi.org/10.1109/TrustCom50675.2020.00169

  32. Megahed, A, Han, Q(2022) Identify videos with facial manipulations based on convolution neural network and dynamic texture. Multimed Tools Appl 1–26

  33. Nirkin, Y, Keller, Y, Hassner, T (2019) Fsgan: Subject agnostic face swapping and reenactment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7184–7193

  34. Prajwal, K, Mukhopadhyay, R, Namboodiri, VP, Jawahar, C (2020) A lip sync expert is all you need for speech to lip generation in the wild. In: Proceedings of the 28th ACM International Conference on Multimedia, p. 484–492

  35. Pun C-M, Liu B, Yuan X-C (2016) Multi-scale noise estimation for image splicing forgery detection. J Vis Commun Image Represent 38:195–206. https://doi.org/10.1016/j.jvcir.2016.03.005

    Article  Google Scholar 

  36. Rahmouni, N, Nozick, V, Yamagishi, J, Echizen, I(2017) Distinguishing computer graphics from natural images using convolution neural networks. In: 2017 IEEE Workshop on Information Forensics and Security (WIFS), p. 1–6. IEEE

  37. Rossler, A, Cozzolino, D, Verdoliva, L, Riess, C, Thies, J, Niesner, M (2018) Faceforensics: A large-scale video dataset for forgery detection in human faces. arXiv:1803.09179

  38. Rossler, A, Cozzolino, D, Verdoliva, L, Riess, C, Thies, J, Niesner, M (2019) Faceforensics++: Learning to detect manipulated facial images. In: Proceedings of the IEEE International Conference on Computer Vision, p. 1–11

  39. Sabir, E, Cheng, J, Jaiswal, A, AbdAlmageed, W, Masi, I, Natarajan, P(2019) Recurrent convolutional strategies for face manipulation detection in videos. Interfaces (GUI) 3(1)

  40. Shaik KB, Ganesan P, Kalist V, Sathish B, Jenitha JMM (2015) Comparative study of skin color detection and segmentation in hsv and ycbcr color space. Procedia Computer Science 57:41–48

    Article  Google Scholar 

  41. Suganthi S, Ayoobkhan MUA, Bacanin N, Venkatachalam K, Štěpán H, Pavel T et al (2022) Deep learning model for deep fake face recognition and detection. PeerJ Comput Sci 8:881

    Article  Google Scholar 

  42. Thies, J, Zollhofer, M, Stamminger, M, Theobalt, C, Niesner, M(2016) Face2face: Real-time face capture and reenactment of rgb videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 2387–2395

  43. Wang, G, Zhou, J, Wu, Y(2020) Exposing deep-faked videos by anomalous co-motion pattern detection. arXiv:2008.04848

  44. Wu, X, Xie, Z, Gao, Y, Xiao, Y (2020) Sstnet: Detecting manipulated faces through spatial, steganalysis and temporal features. In: ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 2952–2956. IEEE

  45. Yang, X, Li, Y, Lyu, S (2019) Exposing deep fakes using inconsistent head poses. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 8261–8265. IEEE

  46. Zhang Q, Lu W, Weng J (2016) Joint image splicing detection in dct and contourlet transform domain. J Vis Commun Image Represent 40:449–458. https://doi.org/10.1016/j.jvcir.2016/07.013

    Article  Google Scholar 

  47. Zhou, P, Han, X, Morariu, VI, Davis, LS(2017) Two-stream neural networks for tampered face detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), p. 1831–1839. IEEE

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China [grant numbers 61771168, 61471141, 61361166006, 61571018, and 61531003]; Key Technology Program of Shenzhen, China, [grant number JSGG20160427185010977]; Basic Research Project of Shenzhen, China [grant number JCYJ20150513151706561].

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Qi Han.

Ethics declarations

Conflicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Megahed, A., Han, Q. & Fadl, S. Exposing deepfake using fusion of deep-learned and hand-crafted features. Multimed Tools Appl 83, 26797–26817 (2024). https://doi.org/10.1007/s11042-023-16329-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16329-2

Keywords