skip to main content
10.1145/3595916.3626426acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Multi-Task Self-Blended Images for Face Forgery Detection

Published: 01 January 2024 Publication History

Abstract

Deepfake detection has attracted extensive attention due to widespread forged images on social media. Recently, self-supervised learning (SSL) based Deepfake detection approaches have outperformed supervised methods in terms of model generalization. However, we notice that most SSL-based methods do not take the manipulation strength levels of synthesized forgery samples into consideration according to different synthesis parameters and result in suboptimal detection performances. To address this issue, we introduce several auxiliary losses to the state-of-the-art SSL-based method based on different synthesis sub-tasks during data generation by inferring their synthesis parameters where the ground-truth labels are obtained from the synthesis pipeline for free. With comprehensive evaluations on various benchmarks, our approach has achieved noticeable performance improvement. Specifically, for the cross-dataset evaluation, the proposed approach outperforms the state-of-the-art method in terms of AUC on various datasets with improvements of 3.4%, 1.47%, 1.56%, and 1.3% on the CDF, DFDC, DFDCP, and FFIW datasets and achieves competitive performance on the DFD dataset. This further demonstrates the effectiveness of the proposed approach in its generalization ability.

References

[1]
Weiming Bai, Yufan Liu, Zhipeng Zhang, Bing Li, and Weiming Hu. 2023. AUNet: Learning Relations Between Action Units for Face Forgery Detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[2]
Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. 2022. Self-supervised Learning of Adversarial Examples: Towards Good Generalizations for DeepFake Detections. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[3]
Weixuan Chen and Daniel McDuff. 2018. Deepphys: Video-based physiological measurement using convolutional attention networks. In European Conference on Computer Vision (ECCV).
[4]
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[5]
Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. 2020. The DeepFake Detection Challenge Dataset. arXiv preprint arXiv:2006.07397. arXiv:2006.07397 [cs.CV]
[6]
Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv preprint arXiv:1910.08854. arXiv:1910.08854 [cs.CV]
[7]
Shichao Dong, Jin Wang, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge. 2023. Implicit Identity Leakage: The Stumbling Block to Improving Deepfake Detection Generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[8]
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2021. Sharpness-aware Minimization for Efficiently Improving Generalization. International Conference on Learning Representations (ICLR).
[9]
Hui Guo, Shu Hu, Xin Wang, Ming-Ching Chang, and Siwei Lyu. 2022. Eyes Tell All: Irregular Pupil Shapes Reveal GAN-Generated Faces. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
[10]
Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. 2022. Leveraging real talking faces via self-supervision for robust forgery detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[11]
Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2021. Lips Don’t Lie: A Generalisable and Robust Approach To Face Forgery Detection. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[12]
Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. 2020. Face X-Ray for More General Face Forgery Detection. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[13]
Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020. Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[14]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. 2017. Focal Loss for Dense Object Detection. IEEE/CVF International Conference on Computer Vision (ICCV).
[15]
Iacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt, and Wael AbdAlmageed. 2020. Two-branch recurrent network for isolating deepfakes in videos. European Conference on Computer Vision (ECCV).
[16]
Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. In ACM SIGGRAPH 2003 Papers.
[17]
Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Niessner. 2019. FaceForensics++: Learning to Detect Manipulated Facial Images. IEEE/CVF International Conference on Computer Vision (ICCV).
[18]
Mohammad Shahid, Shang-Fu Chen, Yu-Ling Hsu, Yung-Yao Chen, Yi-Ling Chen, and Kai-Lung Hua. 2023. Forest Fire Segmentation via Temporal Transformer from Aerial Images. Forests.
[19]
Mohammad Shahid, Isabel Chien, Wannaporn Sarapugdi, Lili Miao, and Kai-Lung Hua. 2020. Deep spatial-temporal networks for flame detection. Multimedia Tools and Applications 80, 35297 – 35318.
[20]
Mohammad Shahid and Kai-Lung Hua. 2021. Fire Detection using Transformer Network. Proceedings of the 2021 International Conference on Multimedia Retrieval.
[21]
Mohammad Shahid, John Jethro Virtusio, Yu-Hsien Wu, Yung-Yao Chen, Muhammad Tanveer, Khan Muhammad, and Kai-Lung Hua. 2021. Spatio-Temporal Self-Attention Network for Fire Detection and Segmentation in Video Surveillance. IEEE Access PP, 1–1.
[22]
Kaede Shiohara and Toshihiko Yamasaki. 2022. Detecting Deepfakes With Self-Blended Images. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[23]
Chang-Sung Sung, Jun-Cheng Chen, and Chu-Song Chen. 2023. Hearing and Seeing Abnormality: Self-Supervised Audio-Visual Mutual Learning for Deepfake Detection. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
[24]
Mingxing Tan and Quoc Le. 2019. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International Conference on Machine Learning (ICML).
[25]
Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred Neural Rendering: Image Synthesis Using Neural Textures. ACM Transaction on Graphics, 12 pages.
[26]
Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Niessner. 2016. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[27]
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.Journal of machine learning research 9, 11.
[28]
Ping Wang, Kunlin Liu, Wenbo Zhou, Hang Zhou, Honggu Liu, Weiming Zhang, and Nenghai Yu. 2022. ADT: Anti-Deepfake Transformer. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
[29]
Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, and Houqiang Li. 2023. AltFreezing for More General Video Face Forgery Detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[30]
Xi Wu, Zhen Xie, YuTao Gao, and Yu Xiao. 2020. SSTNet: Detecting Manipulated Faces Through Spatial, Steganalysis and Temporal Features. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
[31]
Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. 2021. Multi-Attentional Deepfake Detection. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[32]
Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia. 2021. Learning Self-Consistency for Deepfake Detection. IEEE/CVF International Conference on Computer Vision (ICCV).
[33]
Xiaohui Zhao, Yang Yu, Rongrong Ni, and Yao Zhao. 2022. Exploring Complementarity of Global and Local Spatiotemporal Information for Fake Face Video Detection. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
[34]
Yinglin Zheng, Jianmin Bao, Dong Chen, Ming Zeng, and Fang Wen. 2021. Exploring Temporal Coherence for More General Video Face Forgery Detection. IEEE/CVF International Conference on Computer Vision (ICCV).
[35]
Tianfei Zhou, Wenguan Wang, Zhiyuan Liang, and Jianbing Shen. 2021. Face forensics in the wild. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Cited By

View all
  • (2024)Improving Sequential DeepFake Detection with Local information enhancementProceedings of the 6th ACM International Conference on Multimedia in Asia10.1145/3696409.3700276(1-1)Online publication date: 3-Dec-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MMAsia '23: Proceedings of the 5th ACM International Conference on Multimedia in Asia
December 2023
745 pages
ISBN:9798400702051
DOI:10.1145/3595916
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 January 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Deepfake Detection
  2. Face Forgery Detection
  3. Multi-Task
  4. Self-Supervised Learning

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

MMAsia '23
Sponsor:
MMAsia '23: ACM Multimedia Asia
December 6 - 8, 2023
Tainan, Taiwan

Acceptance Rates

Overall Acceptance Rate 59 of 204 submissions, 29%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)127
  • Downloads (Last 6 weeks)9
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Improving Sequential DeepFake Detection with Local information enhancementProceedings of the 6th ACM International Conference on Multimedia in Asia10.1145/3696409.3700276(1-1)Online publication date: 3-Dec-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media