Skip to main content
SpringerLink
Log in

Potential Attacks of DeepFake on eKYC Systems and Remedy for eKYC with DeepFake Detection Using Two-Stream Network of Facial Appearance and Motion Features

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

eKYC, or electronic Know Your Customer, is an electronic or online customer identification allowing banks to completely identify customers online, simplifying paperwork or biometrics verification without having to meet face-to-face at the transaction office like traditional KYC. This new promising trend is becoming widely used in banks and various online services. Thus, eKYC can be a potential target for biometrics spoofing, including possible attacks by deepfake media, such as face swap or manipulation. In this paper, we introduce a potential attack of deepfake photos or videos on eKYC by swapping and manipulating faces between source and target faces. We then propose a novel method to detect fake videos of human faces using a two-stream network. We intend to utilize both the visual appearance and motion of a face in a sequence of pictures. We also propose to encode motion in an image sequence as an average photo which is called a motion-encoded image. Experiments on FaceForensics\(^{++}\) show that using only a small number of frames in a video, our method can provide high accuracy in both frame-level and video-level prediction. With only 10 frames, our solution achieves the accuracy of 0.9326 and 0.9357 for the frame and video-level accuracy, respectively. If we use up to 50 frames per video, the accuracy of our method increases to 0.9371 and 0.9486 at the frame and video level. Finally, we propose to augment the security of current eKYC systems with deepfake detection. Our solution for deepfake detection can be a promising method to provide extra protection for eKYC solutions with face registration and authentication.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Faceswap. https://github.com/MarekKowalski/FaceSwap (2017). Accessed 18 Jul 2022.

  2. Deepfake. https://github.com/deepfakes/faceswap (2018). Accessed 18 Jul 2022.

  3. Terrifying high-tech porn: Creepy ‘deepfake’ videos are on the rise. https://www.foxnews.com/tech/terrifying-high-tech-porn-creepy-deepfake-videos-are-on-the-rise (2018). Accessed 18 Jul 2022.

  4. Agarwal S, Farid H, Fried O, Agrawala M. Detecting deep-fake videos from phoneme-viseme mismatches. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops (2020).

  5. Agarwal S, Farid H, Fried O, Agrawala M. Detecting deep-fake videos from phoneme-viseme mismatches. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW). 2020; p. 2814–22.

  6. Agarwal S, Farid H, Gu Y, He M, Nagano K, Li H. Protecting world leaders against deep fakes. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops (2019).

  7. Bappy JH, Simons C, Nataraj L, Manjunath BS, Roy-Chowdhury AK. Hybrid lstm and encoder–decoder architecture for detection of image forgeries. IEEE Trans Image Process. 2019;28(7):3286–300.

    Article  MathSciNet  MATH  Google Scholar 

  8. Bhattacharjee S, Mohammadi A, Anjos A, Marcel S. Recent advances in face presentation attack detection. In: Marcel S, Nixon MS, Fiérrez J, Evans NWD, editors. Handbook of biometric anti-spoofing—presentation attack detection. Advances in computer vision and pattern recognition. 2nd ed. Berlin: Springer; 2019. p. 207–28.

    Google Scholar 

  9. Chen H, Hu G, Lei Z, Chen Y, Robertson NM, Li SZ. Attention-based two-stream convolutional networks for face spoofing detection. IEEE Trans Inf Forensics Secur. 2020;15:578–93. https://doi.org/10.1109/TIFS.2019.2922241.

    Article  Google Scholar 

  10. Chollet F. Xception: deep learning with depthwise separable convolutions. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). Honolulu, HI: IEEE; 2017. p. 1800–7.

  11. Chugh K, Gupta P, Dhall A, Subramanian R. Not made for each other- audio-visual dissonance-based deepfake detection and localization. In: Proceedings of the 28th ACM international conference on multimedia; 2020.

  12. Costa-Pazo A, Vázquez-Fernández E, Alba-Castro JL, González-Jiménez D. Challenges of face presentation attack detection in real scenarios. In: Marcel S, Nixon MS, Fiérrez J, Evans NWD, editors. Handbook of biometric anti-spoofing—presentation attack detection. Advances in computer vision and pattern recognition. 2nd ed. Berlin: Springer; 2019. p. 247–66.

    Google Scholar 

  13. Davletshin A. https://github.com/ntech-lab/deepfakedetection-challenge (2020).

  14. Do T, Tran M, Nguyen HH, Tran M. Potential threat of face swapping to ekyc with face registration and augmented solution with deepfake detection. In: Dang TK, Küng J, Chung TM, Takizawa M, editors. Future data and security engineering—8th international conference, FDSE 2021, virtual event, November 24–26, 2021, Proceedings. Lecture notes in computer science, vol. 13076. Springer; 2021. p. 293–307. https://doi.org/10.1007/978-3-030-91387-8_19.

  15. Dolhansky B, Bitton J, Pflaum B, Lu J, Howes R, Wang M, Canton-Ferrer C. The deepfake detection challenge dataset. CoRR arXiv:abs/2006.07397 (2020)

  16. Haliassos A, Vougioukas K, Petridis S, Pantic M. Lips don’t lie: a generalisable and robust approach to face forgery detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR); 2021. p. 5039–49.

  17. Hernandez-Ortega J, Fiérrez J, Morales A, Galbally J. Introduction to face presentation attack detection. In: Marcel S, Nixon MS, Fiérrez J, Evans NWD, editors. Handbook of biometric anti-spoofing—presentation attack detection. Advances in computer vision and pattern recognition. 2nd ed. Berlin: Springer; 2019. p. 187–206.

    Google Scholar 

  18. Hu J, Liao X, Liang J, Zhou W, Qin Z. FInfer: Frame inference-based deepfake detection for high-visual-quality videos. In: Proceedings of the AAAI conference on artificial intelligence 2022; 2022.

  19. Joseph Z, Nyirenda C. Deepfake detection using a two-stream capsule network. In: 2021 IST-Africa conference (IST-Africa); 2021. p. 1–8.

  20. Kim H, Garrido P, Tewari A, Xu W, Thies J, Nießner M, Pérez P, Richardt C, Zollöfer M, Theobalt C. Deep video portraits. ACM Trans Graph (TOG). 2018;37(4):163.

    Article  Google Scholar 

  21. Li L, Bao J, Zhang T, Yang H, Chen D, Wen F, Guo B. Face X-ray for more general face forgery detection. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR); 2020. p. 5000–9.

  22. Li Y, Chang MC, Lyu S. In ictu oculi: exposing ai created fake videos by detecting eye blinking. In: 2018 IEEE international workshop on information forensics and security (WIFS); 2018. p. 1–7.

  23. Li Y, Chang MC, Lyu S. In ictu oculi: Exposing ai created fake videos by detecting eye blinking. In: 2018 IEEE international workshop on information forensics and security (WIFS); (2018). p. 1–7. https://doi.org/10.1109/WIFS.2018.8630787.

  24. Li Y, Lyu S. Exposing deepfake videos by detecting face warping artifacts. arXiv:abs/1811.00656 (2019)

  25. Marcel S, Nixon MS, Fiérrez J, Evans NWD, editors. Handbook of Biometric Anti-Spoofing—Presentation Attack Detection, Second Edition. Advances in Computer Vision and Pattern Recognition. Springer; 2019.

  26. Mirsky Y, Lee W. The creation and detection of deepfakes: a survey. ACM Comput Surv. New York, 2021;54(1):1–41. https://doi.org/10.1145/3425780.

  27. Mondal PC, Deb R, Huda MN. Transaction authorization from know your customer (kyc) information in online banking. In: 2016 9th international conference on electrical and computer engineering (ICECE); 2016. p. 523–6.

  28. Nguyen HH, Yamagishi J, Echizen I. Capsule-forensics: using capsule networks to detect forged images and videos. In: ICASSP 2019—2019 IEEE international conference on acoustics. speech and signal processing (ICASSP). Brighton, United Kingdom, May. IEEE; 2019. p. 2307–11.

  29. Nguyen K, Dinh D, Do MN, Tran M. Anomaly detection in traffic surveillance videos with gan-based future frame prediction. In: Gurrin C, Jónsson BT, Kando N, Schöffmann K, Chen YP, O’Connor NE, editors. Proceedings of the 2020 on international conference on multimedia retrieval, ICMR 2020, Dublin, Ireland, June 8–11, 2020. ACM; 2020. p. 457–63. https://doi.org/10.1145/3372278.3390701.

  30. Nguyen K, Hoang T, Tran M, Le T, Bui N, Do T, Vo-Ho V, Luong Q, Tran M, Nguyen T, Truong T, Nguyen V, Do MN. Vehicle re-identification with learned representation and spatial verification and abnormality detection with multi-adaptive vehicle detectors for traffic video analysis. In: IEEE conference on computer vision and pattern recognition workshops, CVPR workshops 2019, Long Beach, CA, USA, June 16–20, 2019. Computer Vision Foundation/IEEE; 2019. p. 363–72.

  31. Nirkin Y, Keller Y, Hassner T. Fsgan: subject agnostic face swapping and reenactment. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV); 2019.

  32. Perov I, Gao D, Chervoniy N, Liu K, Marangonda S, Umé C, Dpfks M, Facenheim CS, RP L, Jiang J, Zhang S, Wu P, Zhou B, Zhang W. Deepfacelab: a simple, flexible and extensible face swapping framework. Comput Vis Pattern Recognit. 2020.

  33. Raghavendra R, Raja KB, Venkatesh S, Busch C. Transferable deep-cnn features for detecting digital and print-scanned morphed face images. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW); 2017. p. 1822–30.

  34. Rossler A, Cozzolino D, Verdoliva L, Riess C, Thies J, Niessner M. FaceForensics++: Learning to Detect Manipulated Facial Images. In: 2019 IEEE/CVF international conference on computer vision (ICCV). Seoul, Korea (South): IEEE; 2019. p. 1–11.

  35. Sabir E, Cheng J, Jaiswal A, AbdAlmageed W, Masi I, Natarajan P. Recurrent convolutional strategies for face manipulation detection in videos. In: IEEE conference on computer vision and pattern recognition workshops, CVPR workshops 2019, Long Beach, CA, USA, June 16–20, 2019. Computer Vision Foundation/IEEE; 2019. p. 80–7.

  36. Sabour S, Frosst N, Hinton GE. Dynamic routing between capsules. In: Proceedings of the 31st international conference on neural information processing systems. NIPS’17. Red Hook, NY, USA: Curran Associates Inc.; 2017. p. 3859–69.

  37. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCun Y, editors. 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, conference track Proceedings. http://arxiv.org/abs/1409.1556 (2015).

  38. Thies J, Zollhofer M, Stamminger M, Theobalt C, Niessner M. Face2face: real-time face capture and reenactment of rgb videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR); 2016.

  39. Thies J, Zollhöfer M, Nießner M. Deferred neural rendering: image synthesis using neural textures. ACM Trans Graph. 2019;38(4):1–12.

    Article  Google Scholar 

  40. Wang JS. Exploring biometric identification in Fintech applications based on the modified tam. Finan Innov. 2021;7:1–24.

    Article  Google Scholar 

  41. Yang X, Li Y, Lyu S. Exposing deep fakes using inconsistent head poses. In: ICASSP 2019 - 2019 IEEE international conference on acoustics, speech and signal processing (ICASSP); 2019. p. 8261–5.

  42. Yang X, Li Y, Lyu S. Exposing deep fakes using inconsistent head poses. In: ICASSP 2019—2019 IEEE international conference on acoustics, speech and signal processing (ICASSP); 2019. p. 8261–5. https://doi.org/10.1109/ICASSP.2019.8683164.

  43. Zakharov E, Shysheya A, Burkov E, Lempitsky V. Few-shot adversarial learning of realistic neural talking head models. In: 2019 IEEE/CVF international conference on computer vision (ICCV). IEEE; 2019. p. 9458–67.

  44. Zhao H, Zhou W, Chen D, Wei T, Zhang W, Yu N. Multi-attentional deepfake detection. In: IEEE conference on computer vision and pattern recognition, CVPR 2021, virtual, June 19–25, 2021. Computer Vision Foundation/IEEE; 2021. p. 2185–94.

  45. Zhao Z, Wang P, Lu W. Detecting deepfake video by learning two-level features with two-stream convolutional neural network. In: Proceedings of the 2020 6th international conference on computing and artificial intelligence. ICCAI ’20. New York, NY, USA: Association for Computing Machinery; 2020. p. 291–7. https://doi.org/10.1145/3404555.3404564.

  46. Zheng Y, Bao J, Chen D, Zeng M, Wen F. Exploring temporal coherence for more general video face forgery detection. In: Proceedings of the IEEE/CVF international conference on computer vision; 2021. p. 15044–54.

  47. Zhou P, Han X, Morariu VI, Davis LS. Two-stream neural networks for tampered face detection. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW); 2017. p. 1831–9. https://doi.org/10.1109/CVPRW.2017.229.

  48. Zhou P, Han X, Morariu V, Larry Davis PIS. Two-stream neural networks for tampered face detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops; 2017.

  49. Zhu Y, Li Q, Wang J, Xu C, Sun Z. One shot face swapping on megapixels. In: IEEE conference on computer vision and pattern recognition, CVPR 2021, virtual, June 19–25, 2021. Computer Vision Foundation/IEEE; 2021. p. 4834–44.

Download references

Funding

This research is funded by University of Science, VNU-HCM under grantnumber T2022-85.

Author information

Authors and Affiliations

  1. University of Science, VNU-HCM, Ho Chi Minh City, Vietnam

    Trong-Le Do, Mai-Khiem Tran & Minh-Triet Tran

  2. John von Neumann Institute, VNU-HCM, Ho Chi Minh City, Vietnam

    Mai-Khiem Tran & Minh-Triet Tran

  3. Viet Nam National University, Ho Chi Minh City, Vietnam

    Trong-Le Do, Mai-Khiem Tran & Minh-Triet Tran

  4. National Institute of Informatics, Tokyo, Japan

    Huy H. Nguyen

Authors
  1. Trong-Le Do
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mai-Khiem Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huy H. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Minh-Triet Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Minh-Triet Tran.

Ethics declarations

Conflict of interest

All authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Future Data and Security Engineering 2021” guest edited by Tran Khanh Dang.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Do, TL., Tran, MK., Nguyen, H.H. et al. Potential Attacks of DeepFake on eKYC Systems and Remedy for eKYC with DeepFake Detection Using Two-Stream Network of Facial Appearance and Motion Features. SN COMPUT. SCI. 3, 464 (2022). https://doi.org/10.1007/s42979-022-01364-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01364-x

Keywords

Search

Navigation