Abstract
The advantages of deepfakes in many applications are counterbalanced by their malicious use, for example, in reply-attacks against a biometric system, identification evasion, and people harassment, when they are widespread in social networks and chatting platforms (cyberbullying) as recently documented in newspapers. Due to its “arms-race” nature, deepfake detection systems are often trained on a certain class of deepfakes and showed their limits on never-seen-before classes. In order to shed some light on this problem, we explore the benefits of a multi-modal deepfake detection system. We adopted simple fusion rules, which showed their effectiveness in many applications, for example, biometric recognition, to exploit the complementary of different individual classifiers, and derive some possible guidelines for the designer.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: MesoNet: a compact facial video forgery detection network, pp. 1–7 (2018). https://doi.org/10.1109/WIFS.2018.8630761
Bekci, B., Akhtar, Z., Ekenel, H.K.: Cross-dataset face manipulation detection. In: 2020 28th Signal Processing and Communications Applications Conference (SIU), pp. 1–4. IEEE (2020)
Bonettini, N., Cannas, E.D., Mandelli, S., Bondi, L., Bestagini, P., Tubaro, S.: Video face manipulation detection through ensemble of CNNs. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 5012–5019 (2021). https://doi.org/10.1109/ICPR48806.2021.9412711
Chugh, T., Jain, A.K.: Fingerprint spoof detector generalization. IEEE Trans. Inf. Forensics Secur. 16, 42–55 (2021). https://doi.org/10.1109/TIFS.2020.2990789
Cozzolino, D., Gragnaniello, D., Poggi, G., Verdoliva, L.: Towards universal GAN image detection. In: 2021 International Conference on Visual Communications and Image Processing (VCIP), pp. 1–5 (2021). https://doi.org/10.1109/VCIP53242.2021.9675329
Cozzolino, D., Verdoliva, L.: Noiseprint: a CNN-based camera model fingerprint. IEEE Trans. Inf. Forensics Secur. 15, 144–159 (2020). https://doi.org/10.1109/TIFS.2019.2916364
Didaci, L., Marcialis, G.L., Roli, F.: Analysis of unsupervised template update in biometric recognition systems. Pattern Recogn. Lett. 37, 151–160 (2014). https://doi.org/10.1016/j.patrec.2013.05.021
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., Ferrer, C.C.: The DeepFake Detection Challenge (DFDC) preview dataset (2019)
Güera, D., Delp, E.J.: Deepfake video detection using recurrent neural networks. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 (2018). https://doi.org/10.1109/AVSS.2018.8639163
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hernandez-Ortega, J., Tolosana, R., Fierrez, J., Morales, A.: DeepFakes detection based on heart rate estimation: single- and multi-frame, pp. 255–273. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-87664-7_12
Jain, A., Nandakumar, K., Ross, A.: Score normalization in multimodal biometric systems. Pattern Recogn. 38(12), 2270–2285 (2005). https://doi.org/10.1016/j.patcog.2005.01.012
Korshunov, P., Marcel, S.: DeepFakes: a new threat to face recognition? Assessment and detection. arXiv preprint arXiv:1812.08685 (2018)
Li, Y., Chang, M.C., Lyu, S.: In ictu oculi: exposing AI created fake videos by detecting eye blinking. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–7 (2018). https://doi.org/10.1109/WIFS.2018.8630787
Li, Y., Lyu, S.: Exposing DeepFake videos by detecting face warping artifacts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019
Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-DF: a large-scale challenging dataset for DeepFake forensics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020
Punyani, P., Kumar, A.: Evaluation of fusion at different levels for face recognition. In: 2017 International Conference on Computing, Communication and Automation (ICCCA), pp. 1052–1055, May 2017. https://doi.org/10.1109/CCAA.2017.8229950
Rana, M.S., Sung, A.H.: DeepfakeStack: a deep ensemble-based learning technique for deepfake detection. In: 2020 7th IEEE CSCloud/2020 6th IEEE EdgeCom, pp. 70–75 (2020). https://doi.org/10.1109/CSCloud-EdgeCom49738.2020.00021
Roli, F., Giacinto, G., Vernazza, G.: Methods for designing multiple classifier systems. In: Kittler, J., Roli, F. (eds.) MCS 2001. LNCS, vol. 2096, pp. 78–87. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-48219-9_8
Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niessner, M.: FaceForensics++: learning to detect manipulated facial images. In: ICCV 2019, pp. 1–11 (2019). https://doi.org/10.1109/ICCV.2019.00009
Sun, F., Zhang, N., Xu, P., Song, Z.: Deepfake detection method based on cross-domain fusion. Secur. Commun. Netw. 2021, 2482942 (2021). https://doi.org/10.1155/2021/2482942
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2Face: real-time face capture and reenactment of RGB videos. Commun. ACM 62(1), 96–104 (2018). https://doi.org/10.1145/3292039
Tolosana, R., Romero-Tapiador, S., Fierrez, J., Vera-Rodriguez, R.: DeepFakes evolution: analysis of facial regions and fake detection performance. In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12665, pp. 442–456. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68821-9_38
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., Ortega-Garcia, J.: Deepfakes and beyond: a survey of face manipulation and fake detection. Inf. Fusion 64, 131–148 (2020). https://doi.org/10.1016/j.inffus.2020.06.014
Vishi, K., Mavroeidis, V.: An evaluation of score level fusion approaches for fingerprint and finger-vein biometrics (2018)
Yu, P., Xia, Z., Fei, J., Lu, Y.: A survey on deepfake video detection. IET Biom. 10(6), 607–624 (2021). https://doi.org/10.1049/bme2.12031
Zhao, L., Zhang, M., Ding, H., Cui, X.: MFF-Net: deepfake detection network based on multi-feature fusion. Entropy 23(12), 1692 (2021). https://doi.org/10.3390/e23121692
Zhao, Z., Wang, P., Lu, W.: Multi-layer fusion neural network for deepfake detection. Int. J. Digit. Crime Forensics (IJDCF) 13(4), 26–39 (2021). https://doi.org/10.4018/IJDCF.20210701.oa3
Acknowledgment
This work is supported by the Italian Ministry of Education, University and Research (MIUR) within the PRIN2017 - BullyBuster - A framework for bullying and cyberbullying action detection by computer vision and artificial intelligence methods and algorithms (CUP: F74I19000370001).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Concas, S. et al. (2022). Experimental Results on Multi-modal Deepfake Detection. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds) Image Analysis and Processing – ICIAP 2022. ICIAP 2022. Lecture Notes in Computer Science, vol 13232. Springer, Cham. https://doi.org/10.1007/978-3-031-06430-2_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-06430-2_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06429-6
Online ISBN: 978-3-031-06430-2
eBook Packages: Computer ScienceComputer Science (R0)