skip to main content
10.1145/3581783.3613752acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

BiFPro: A Bidirectional Facial-data Protection Framework against DeepFake

Published: 27 October 2023 Publication History

Abstract

The rapid progress of the DeepFake technique has caused severe privacy problems. Thus protecting facial data against DeepFake becomes an urgent requirement. Face protection can be regarded as a bidirectional process: Face-out-detection (FOD) and Face-in-forensics (FIF). For FOD, the detectability should be satisfied when using the protected face to replace other faces. For FIF, traceability should be guaranteed when the protected face is replaced by others. For this, we propose a Bidirectional Facial-data Protection Framework (BiFPro) to protect face data comprehensively. This framework is composed of three main parts: Watermarking embedding, Face-out-detection (FOD) and Face-in-forensics (FIF). For the FOD case, we ensure the vulnerability of the original face by embedding fragile watermarking. Once the protected facial image is used to replace other faces, the watermarking information will be corrupted in the synthesized face images which can be used to detect the authenticity of the protected facial images. As for the FIF case, we guarantee the traceability of the protected face image by embedding robust watermarking, with which the fake faces can be traced with the reserved watermarking even after the face is swapped. Experimental results demonstrate that our proposed BiFPro could generate the watermarking which is fragile to FOD and at the same time robust to FIF with an average watermark extraction success rate reaching more than 95% when defending against the four advanced DeepFake techniques. Finally, we hope this work can encourage more initiative countermeasures against DeepFake.

Supplemental Material

MP4 File
Presentation video of mmfp1466

References

[1]
Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. 2018. Mesonet: a compact facial video forgery detection network. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS). IEEE, 1--7.
[2]
Shruti Agarwal, Hany Farid, Tarek El-Gaaly, and Ser-Nam Lim. 2020. Detecting deep-fake videos from appearance and behavior. In 2020 IEEE International Workshop on Information Forensics and Security (WIFS). IEEE, 1--6.
[3]
Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. 2019. Protecting World Leaders Against Deep Fakes. In CVPR workshops, Vol. 1.
[4]
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. 2018. Towards open-set identity preserving face synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6713--6722.
[5]
Mahbuba Begum and Mohammad Shorif Uddin. 2020. Digital image watermarking techniques: a review. Information, Vol. 11, 2 (2020), 110.
[6]
Adrian G Bors and Ioannis Pitas. 1996. Image watermarking using DCT domain constraints. In Proceedings of 3rd IEEE International Conference on Image Processing, Vol. 3. IEEE, 231--234.
[7]
Renwang Chen, Xuanhong Chen, Bingbing Ni, and Yanhao Ge. 2020. SimSwap: An Efficient Framework For High Fidelity Face Swapping. In Proceedings of the 28th ACM International Conference on Multimedia. 2003--2011.
[8]
François Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1251--1258.
[9]
Ingemar J Cox, Joe Kilian, F Thomson Leighton, and Talal Shamoon. 1997. Secure spread spectrum watermarking for multimedia. IEEE transactions on image processing, Vol. 6, 12 (1997), 1673--1687.
[10]
DeepFakes. 2017. DeepFakes Github. http://github.com/deepfakes/faceswap. Accessed 2021-08--25.
[11]
Sefik Emre Eskimez, You Zhang, and Zhiyao Duan. 2021. Speech driven talking face generation from a single image and an emotion condition. IEEE Transactions on Multimedia (2021).
[12]
Han Fang, Dongdong Chen, Qidong Huang, Jie Zhang, Zehua Ma, Weiming Zhang, and Nenghai Yu. 2020. Deep Template-based Watermarking. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, 4 (2020), 1436--1451.
[13]
Han Fang, Zhaoyang Jia, Hang Zhou, Zehua Ma, and Weiming Zhang. 2022. Encoded Feature Enhancement in Watermarking Network for Distortion in Real Scenes. IEEE Transactions on Multimedia (2022).
[14]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
[15]
Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2021. Lips Don't Lie: A Generalisable and Robust Approach To Face Forgery Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5039--5049.
[16]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[17]
Mei Jiansheng, Li Sukang, and Tan Xiaomei. 2009. A digital watermarking algorithm based on DCT and DWT. In Proceedings. The 2009 International Symposium on Web Information Systems and Applications (WISA 2009). Citeseer, 104.
[18]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision. Springer, 694--711.
[19]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017).
[20]
Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4401--4410.
[21]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8110--8119.
[22]
S Katzenbeisser and FAP Petitcolas. 2000. Digital watermarking. Artech House, London, Vol. 2 (2000).
[23]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[24]
Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
[25]
Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. 2017. Fast face-swap using convolutional neural networks. In Proceedings of the IEEE international conference on computer vision. 3677--3685.
[26]
Akash Kumar, Arnav Bhavsar, and Rajesh Verma. 2020. Detecting deepfakes with metric learning. In 2020 8th international workshop on biometrics and forensics (IWBF). IEEE, 1--6.
[27]
Jiaming Li, Hongtao Xie, Jiahong Li, Zhongyuan Wang, and Yongdong Zhang. 2021. Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6458--6467.
[28]
Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. 2020a. Advancing high fidelity identity swapping for forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5074--5083.
[29]
Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. 2020b. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5001--5010.
[30]
Yuezun Li and Siwei Lyu. 2018. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv:1811.00656 (2018).
[31]
Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020c. Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3207--3216.
[32]
Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu. 2021. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 772--781.
[33]
Ming-Yu Liu and Oncel Tuzel. 2016. Coupled generative adversarial networks. Advances in neural information processing systems, Vol. 29 (2016), 469--477.
[34]
Hongtao Lu, Ruiming Shen, and Fu-Lai Chung. 2003. Fragile watermarking scheme for image authentication. Electronics Letters, Vol. 39, 12 (2003), 898--900.
[35]
Yuchen Luo, Yong Zhang, Junchi Yan, and Wei Liu. 2021. Generalizing Face Forgery Detection with High-frequency Features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16317--16326.
[36]
Henrique S Malvar and Dinei AF Florêncio. 2003. Improved spread spectrum: A new modulation technique for robust watermarking. IEEE transactions on signal processing, Vol. 51, 4 (2003), 898--905.
[37]
Iacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt, and Wael AbdAlmageed. 2020. Two-branch recurrent network for isolating deepfakes in videos. In European Conference on Computer Vision. Springer, 667--684.
[38]
Falko Matern, Christian Riess, and Marc Stamminger. 2019. Exploiting visual artifacts to expose deepfakes and face manipulations. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). IEEE, 83--92.
[39]
Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. 2018a. Fsnet: An identity-aware generative model for image-based face swapping. In Asian Conference on Computer Vision. Springer, 117--132.
[40]
Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. 2018b. Rsgan: face swapping and editing using face and hair representation in latent spaces. arXiv preprint arXiv:1804.03447 (2018).
[41]
Minh Ngo, Sezer Karaoglu, and Theo Gevers. 2021. Self-supervised Face Image Manipulation by Conditioning GAN on Face Decomposition. IEEE Transactions on Multimedia (2021).
[42]
Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF international conference on computer vision. 7184--7193.
[43]
Ivan Perov, Daiheng Gao, Nikolay Chervoniy, Kunlin Liu, Sugasa Marangonda, Chris Umé, Mr Dpfks, Carl Shift Facenheim, Luis RP, Jian Jiang, et al. 2020. Deepfacelab: A simple, flexible and extensible face swapping framework. arXiv preprint arXiv:2005.05535 (2020).
[44]
Christine I Podilchuk and Edward J Delp. 2001. Digital watermarking: algorithms and applications. IEEE signal processing Magazine, Vol. 18, 4 (2001), 33--46.
[45]
Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. 2020. Thinking in frequency: Face forgery detection by mining frequency-aware clues. In European Conference on Computer Vision. Springer, 86--103.
[46]
Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2019. Faceforensics: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1--11.
[47]
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. First order motion model for image animation. Advances in Neural Information Processing Systems, Vol. 32 (2019), 7137--7147.
[48]
Matthew Tancik, Ben Mildenhall, and Ren Ng. 2020. Stegastamp: Invisible hyperlinks in physical photographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2117--2126.
[49]
Matthieu Urvoy, Dalila Goudia, and Florent Autrusseau. 2014. Perceptual DFT watermarking with improved detection and robustness to geometrical distortions. IEEE Transactions on Information Forensics and Security, Vol. 9, 7 (2014), 1108--1119.
[50]
Ron G Van Schyndel, Andrew Z Tirkel, and Charles F Osborne. 1994. A digital watermark. In Proceedings of 1st international conference on image processing, Vol. 2. IEEE, 86--90.
[51]
Chengrui Wang and Weihong Deng. 2021. Representative Forgery Mining for Fake Face Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14923--14932.
[52]
Junke Wang, Zuxuan Wu, Jingjing Chen, and Yu-Gang Jiang. 2021b. M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection. arXiv preprint arXiv:2104.09770 (2021).
[53]
Run Wang, Felix Juefei-Xu, Meng Luo, Yang Liu, and Lina Wang. 2021a. FakeTagger: Robust Safeguards against DeepFake Dissemination via Provenance Tracking. In Proceedings of the 29th ACM International Conference on Multimedia. 3546--3555.
[54]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, Vol. 13, 4 (2004), 600--612.
[55]
Eric Wengrowski and Kristin Dana. 2019. Light field messaging with deep photographic steganography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1515--1524.
[56]
Raymond B Wolfgang and Edward J Delp. 1996. A watermark for digital images. In Proceedings of 3rd IEEE International Conference on Image Processing, Vol. 3. IEEE, 219--222.
[57]
Xin Yang, Yuezun Li, and Siwei Lyu. 2019. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 8261--8265.
[58]
Yuankun Yang, Chenyue Liang, Hongyu He, Xiaoyu Cao, and Neil Zhenqiang Gong. 2021. FaceGuard: Proactive Deepfake Detection. arXiv preprint arXiv:2109.05673 (2021).
[59]
Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. 2021. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14448--14457.
[60]
Chaoning Zhang, Philipp Benz, Adil Karjauv, Geng Sun, and In So Kweon. 2020. Udh: Universal deep hiding for steganography, watermarking, and light field messaging. Advances in Neural Information Processing Systems, Vol. 33 (2020), 10223--10234.
[61]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586--595.
[62]
Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. 2021. Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2185--2194.
[63]
Yinglin Zheng, Jianmin Bao, Dong Chen, Ming Zeng, and Fang Wen. 2021. Exploring Temporal Coherence for More General Video Face Forgery Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 15044--15054.
[64]
Dawei Zhou, Nannan Wang, Chunlei Peng, Yi Yu, Xi Yang, and Xinbo Gao. 2021. Towards Multi-domain Face Synthesis via Domain-Invariant Representations and Multi-level Feature Parts. IEEE Transactions on Multimedia (2021).
[65]
Jiren Zhu, Russell Kaplan, Justin Johnson, and Li Fei-Fei. 2018. Hidden: Hiding data with deep networks. In Proceedings of the European conference on computer vision (ECCV). 657--672.
[66]
Xiangyu Zhu, Hao Wang, Hongyan Fei, Zhen Lei, and Stan Z Li. 2021. Face Forgery Detection by 3D Decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2929--2939.

Cited By

View all
  • (2024)Dual-Task Cascaded for Proactive Deepfake Detection Using QPCET WatermarkingPattern Recognition and Computer Vision10.1007/978-981-97-8490-5_10(132-147)Online publication date: 7-Nov-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '23: Proceedings of the 31st ACM International Conference on Multimedia
October 2023
9913 pages
ISBN:9798400701085
DOI:10.1145/3581783
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deepfake
  2. digital watermarking
  3. forensics
  4. traceability

Qualifiers

  • Research-article

Funding Sources

  • Key Research and Development program of Anhui Province
  • Natural Science Foundation of China
  • Ant Group through CCF-Ant Innovative Research Program

Conference

MM '23
Sponsor:
MM '23: The 31st ACM International Conference on Multimedia
October 29 - November 3, 2023
Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)188
  • Downloads (Last 6 weeks)22
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Dual-Task Cascaded for Proactive Deepfake Detection Using QPCET WatermarkingPattern Recognition and Computer Vision10.1007/978-981-97-8490-5_10(132-147)Online publication date: 7-Nov-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media