Skip to main content
Springer Nature Link
Log in

Adapting Pretrained Large-Scale Vision Models for Face Forgery Detection

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14557))

Included in the following conference series:

Abstract

In the evolving digital realm, generative networks have catalyzed an upsurge in deceptive media, encompassing manipulated facial imagery to tampered text, threatening both personal security and societal stability. While specialized detection networks exist for specific forgery types, their limitations in handling diverse online forgeries and resource constraints necessitate a more holistic approach. This paper presents a pioneering effort to efficiently adapt pre-trained large vision models (LVMs) for the critical task of forgery detection, emphasizing face forgery. Recognizing the inherent challenges in bridging pre-training tasks with forgery detection, we introduce a novel parameter-efficient adaptation strategy. Our investigations highlight the imperative of focusing on detailed, local features to discern forgery indicators. Departing from conventional methods, we propose the Detail-Enhancement Adapter (DE-Adapter), inspired by ‘Unsharp Masking’. By leveraging Gaussian convolution kernels and differential operations, the DE-Adapter enhances detailed representations. With our method, we achieved state-of-the-art performance with only 0.3% network adjustment. Especially when the number of training samples is limited, our method far surpasses other methods. Our work also provides a new perspective for the Uni-Vision Large Model, and we call on more fields to design suitable adapting schemes to expand the capabilities of large models instead of redesigning networks from scratch.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: MesoNet: a compact facial video forgery detection network. In: WIFS (2018)

    Google Scholar 

  2. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P.N., Nayar, S.K.: Face swapping: automatically replacing faces in photographs. ACM Trans. Graph 27, 1–8 (2008)

    Article  Google Scholar 

  3. Cai, H., Gan, C., Zhu, L., Han, S.: Tinytl: reduce memory, not parameters for efficient on-device learning. In: NeurIPS (2020)

    Google Scholar 

  4. Chen, S., Yao, T., Chen, Y., Ding, S., Li, J., Ji, R.: Local relation learning for face forgery detection. In: AAAI (2021)

    Google Scholar 

  5. Chen, S., et al.: Adaptformer: adapting vision transformers for scalable visual recognition. In: NeurIPS (2022)

    Google Scholar 

  6. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)

    Google Scholar 

  7. Dhariwal, P., Nichol, A.Q.: Diffusion models beat GANs on image synthesis. In: NeurIPS (2021)

    Google Scholar 

  8. Dolhansky, B., et al.: The deepfake detection challenge (DFDC) dataset. CoRR (2020)

    Google Scholar 

  9. Dong, B., Zhou, P., Yan, S., Zuo, W.: LPT: long-tailed prompt tuning for image classification. CoRR (2022)

    Google Scholar 

  10. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR (2021)

    Google Scholar 

  11. Fridrich, J.J., Kodovský, J.: Rich models for steganalysis of digital images. TIFS 7, 868–882 (2012)

    Google Scholar 

  12. Gao, Y., et al.: High-fidelity and arbitrary face editing. In: CVPR (2021)

    Google Scholar 

  13. Goodfellow, I.J., et al.: Generative adversarial networks. CoRR (2014)

    Google Scholar 

  14. Han, X., Morariu, V., Larry Davis, P.I., et al.: Two-stream neural networks for tampered face detection. In: CVPR Workshop (2017)

    Google Scholar 

  15. He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient transfer learning. In: ICLR (2022)

    Google Scholar 

  16. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR (2022)

    Google Scholar 

  17. He, X., Li, C., Zhang, P., Yang, J., Wang, X.E.: Parameter-efficient fine-tuning for vision transformers. CoRR (2022)

    Google Scholar 

  18. Hu, E.J., et al.: Lora: low-rank adaptation of large language models. In: ICLR (2022)

    Google Scholar 

  19. Huang, H., et al.: Adaptive transformers for robust few-shot cross-domain face anti-spoofing. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13673, pp. 37–54. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19778-9_3

    Chapter  Google Scholar 

  20. Huang, Z., Chan, K.C.K., Jiang, Y., Liu, Z.: Collaborative diffusion for multi-modal face generation and editing. In: CVPR (2023)

    Google Scholar 

  21. Jia, G., et al.: Inconsistency-aware wavelet dual-branch network for face forgery detection. Trans. Biom. Behav. Ident. Sci. 3, 308–319 (2021)

    Article  Google Scholar 

  22. Jia, M., et al.: Visual prompt tuning. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13693, pp. 709–727. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_41

    Chapter  Google Scholar 

  23. Juefei-Xu, F., Boddeti, V.N., Savvides, M.: Local binary convolutional neural networks. In: CVPR (2017)

    Google Scholar 

  24. Kim, K., et al.: Diffface: diffusion-based face swapping with facial guidance. CoRR (2022)

    Google Scholar 

  25. Li, L., et al.: Face x-ray for more general face forgery detection. In: CVPR (2020)

    Google Scholar 

  26. Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for deepfake forensics. In: CVPR (2019)

    Google Scholar 

  27. Liu, H., et al. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In: CVPR (2021)

    Google Scholar 

  28. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: ICCV (2021)

    Google Scholar 

  29. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: CVPR (2022)

    Google Scholar 

  30. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)

    Google Scholar 

  31. Miao, C., Tan, Z., Chu, Q., Liu, H., Hu, H., Yu, N.: F\({}^{\text{2}}\)trans: High-frequency fine-grained transformer for face forgery detection. TIFS (2023)

    Google Scholar 

  32. Nguyen, H.H., Fang, F., Yamagishi, J., Echizen, I.: Multi-task learning for detecting and segmenting manipulated facial images and videos. In: BTAS (2019)

    Google Scholar 

  33. Pan, J., Lin, Z., Zhu, X., Shao, J., Li, H.: St-adapter: parameter-efficient image-to-video transfer learning. In: NeurIPS (2022)

    Google Scholar 

  34. Paszke, A., et al.: Automatic differentiation in pytorch (2017)

    Google Scholar 

  35. Qian, Y., Yin, G., Sheng, L., Chen, Z., Shao, J.: Thinking in frequency: face forgery detection by mining frequency-aware clues. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV 2022. LNCS, vol. 12357, pp. 86–103. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_6

    Chapter  Google Scholar 

  36. Rössler, A., Cet al.: Faceforensics++: learning to detect manipulated facial images. In: ICCV (2019)

    Google Scholar 

  37. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: IJCV (2020)

    Google Scholar 

  38. Shi, Z., Chen, Y., Gavves, E., Mettes, P., Snoek, C.G.M.: Unsharp mask guided filtering. TIP 30, 7472–7485 (2021)

    Google Scholar 

  39. Wang, C., Deng, W.: Representative forgery mining for fake face detection. In: CVPR (2021)

    Google Scholar 

  40. Wang, J., et al.: M2tr: multi-modal multi-scale transformers for deepfake detection. In: ICMR (2022)

    Google Scholar 

  41. Wang, Y., Xie, H., Xing, M., Wang, J., Zhu, S., Zhang, Y.: Detecting tampered scene text in the wild. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13688, pp. 215–232. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19815-1_13

    Chapter  Google Scholar 

  42. Woo, S., et al.: Convnext v2: co-designing and scaling convnets with masked autoencoders. CoRR (2023)

    Google Scholar 

  43. Yao, G., et al.: One-shot face reenactment using appearance adaptive normalization. In: AAAI (2021)

    Google Scholar 

  44. Zhao, H., Wei, T., Zhou, W., Zhang, W., Chen, D., Yu, N.: Multi-attentional deepfake detection. In: CVPR (2021)

    Google Scholar 

  45. Zhou, P., Han, X., Morariu, V.I., Davis, L.S.: Two-stream neural networks for tampered face detection. In: CVPR Workshop (2017)

    Google Scholar 

  46. Zi, B., Chang, M., Chen, J., Ma, X., Jiang, Y.: Wilddeepfake: a challenging real-world dataset for deepfake detection. In: ACM MM (2020)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part by NSFC (62376156, 62322113), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), and the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China

    Lantao Wang & Chao Ma

Authors
  1. Lantao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chao Ma .

Editor information

Editors and Affiliations

  1. University of Amsterdam, Amsterdam, The Netherlands

    Stevan Rudinac

  2. Delft University of Technology, Delft, The Netherlands

    Alan Hanjalic

  3. Delft University of Technology, Delft, The Netherlands

    Cynthia Liem

  4. University of Amsterdam, Amsterdam, The Netherlands

    Marcel Worring

  5. Reykjavik University, Reykjavik, Iceland

    Björn Þór Jónsson

  6. Microsoft Research Lab – Asia, Beijing, China

    Bei Liu

  7. The University of Tokyo, Tokyo, Japan

    Yoko Yamakata

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, L., Ma, C. (2024). Adapting Pretrained Large-Scale Vision Models for Face Forgery Detection. In: Rudinac, S., et al. MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14557. Springer, Cham. https://doi.org/10.1007/978-3-031-53302-0_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-53302-0_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-53301-3

  • Online ISBN: 978-3-031-53302-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics