research-article

Multi-modal Multi-layer Fusion Network with Average Binary Center Loss for Face Anti-spoofing

Authors:

Shengchuan Zhang,

Baochang ZhangAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 48 - 56

https://doi.org/10.1145/3343031.3351001

Published: 15 October 2019 Publication History

Abstract

Face anti-spoofing detection is critical to guarantee the security of biometric face recognition systems. Despite extensive advances in facial anti-spoofing based on single-model image, little work has been devoted to multi-modal anti-spoofing, which is however widely encountered in real-world scenarios. Following the recent progress, this paper mainly focuses on multi-modal face anti-spoofing and aims to solve the following two challenges: (1) how to effectively fuse multi-modal information; and (2) how to effectively learn distinguishable features despite single cross-entropy loss. We propose a novel Multi-modal Multi-layer Fusion Convolutional Neural Network (mmfCNN), which targets at finding a discriminative model for recognizing the subtle differences between live and spoof faces. The mmfCNN can fully use different information provided by diverse modalities, which is based on a weight-adaptation aggregation approach. Specifically, we utilize a multi-layer fusion model to further aggregate the features from different layers, which fuses the low-, mid- and high-level information from different modalities in a unified framework. Moreover, a novel Average Binary Center (ABC) loss is proposed to maximize the dissimilarity between the features of live and spoof faces, which helps to stabilize the training to generate a robust and discriminative model. Extensive experiments conducted on the CISIA-SURF and 3DMAD datasets verify the significance and generalization capability of the proposed method for the face anti-spoofing task. Code is available at: https://github.com/SkyKuang/Face-anti-spoofing.

References

[1]

Yousef Atoum, Yaojie Liu, Amin Jourabloo, and Xiaoming Liu. 2018. Face Anti-Spoofing Using Patch and Depth-Based CNNs. In Proceedings of IJCB .

[2]

Zinelabidine Boulkenafet, Jukka Komulainen, and Abdenour Hadid. 2015. Face anti-spoofing based on color texture analysis. In Proceedings of the ICIP) .

Digital Library

[3]

Zinelabidine Boulkenafet, Jukka Komulainen, and Abdenour Hadid. 2016. Face spoofing detection using colour texture analysis. IEEE Trans. on Information Forensics and Security (2016).

Digital Library

[4]

Zinelabidine Boulkenafet, Jukka Komulainen, and Abdenour Hadid. 2017. Face antispoofing using speeded-up robust features and fisher vector encoding. IEEE Signal Processing Letters (2017).

[5]

Ivana Chingovska, André Anjos, and Sébastien Marcel. 2012. On the effectiveness of local binary patterns in face anti-spoofing. In Proceedings of the BIOSIG) .

[6]

Tiago de Freitas Pereira, André Anjos, José Mario De Martino, and Sébastien Marcel. 2012. LBP- TOP based countermeasure against face spoofing attacks. In Asian Conference on Computer Vision . Springer, 121--132.

[7]

Wen Di, Han Hu, and Anil K. Jain. 2015. Face Spoof Detection With Image Distortion Analysis. IEEE Trans. on Information Forensics & Security (2015).

[8]

Sidney K. D'Mello and Jacqueline Kory. 2015. A Review and Meta-Analysis of Multimodal Affect Detection Systems. Comput. Surveys (2015).

[9]

Taiamiti Edmunds and Alice Caplier. 2018. Motion-based countermeasure against photo and video spoofing attacks in face recognition. Journal of Visual Communication and Image Representation, Vol. 50 (2018), 314--332.

[10]

Nesli Erdogmus and Sébastien Marcel. 2013. Spoofing in 2d face recognition with 3d masks and anti-spoofing with kinect. In 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS). IEEE, 1--6.

[11]

Litong Feng, Lai-Man Po, Yuming Li, Xuyuan Xu, Fang Yuan, Terence Chun-Ho Cheung, and Kwok-Wai Cheung. 2016. Integration of image quality and motion cues for face anti-spoofing: A neural network approach. Journal of Visual Communication and Image Representation (2016).

[12]

Javier Galbally, Sébastien Marcel, and Julian Fierrez. 2017. Biometric Antispoofing Methods: A Survey in Face Recognition. IEEE Access (2017).

[13]

Reza Shoja Ghiass, Ognjen Arandjelovic, Abdelhakim Bendada, and Xavier Maldague. 2014. Infrared face recognition: A comprehensive review of methodologies and databases. Pattern Recognition (2014).

[14]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the CVPR .

[15]

Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7132--7141.

[16]

Klaus Kollreider, Hartwig Fronthaler, Maycel Isaac Faraj, and Josef Bigun. 2007. Real-time face detection and motion analysis with application in "liveness" assessment. IEEE Trans. on Information Forensics and Security (2007).

[17]

Jukka Komulainen, Abdenour Hadid, and Matti Pietik"ainen. 2013. Context based face anti-spoofing. In Proceedings of the BTAS .

[18]

Dominic P Laage and Maria T Laage. 2005. Payment instrument authorization technique. US Patent 6,931,382.

[19]

Haoliang Li, Peisong He, Shiqi Wang, Anderson Rocha, Xinghao Jiang, and Alex C Kot. 2018. Learning generalized deep feature representation for face anti-spoofing. IEEE Trans. on Information Forensics and Security (2018).

[20]

Jiangwei Li, Yunhong Wang, Tieniu Tan, and Anil K Jain. 2004. Live face detection based on the analysis of fourier spectra. In Biometric Technology for Human Identification, Vol. 5404. International Society for Optics and Photonics, 296--304.

[21]

Lei Li, Xiaoyi Feng, Zinelabidine Boulkenafet, Zhaoqiang Xia, Mingming Li, and Abdenour Hadid. 2016. An original face anti-spoofing approach using partial convolutional neural network. In Procedding IPTA) .

[22]

Stan Z Li, RuFeng Chu, ShengCai Liao, and Lun Zhang. 2007. Illumination invariant face recognition using near-infrared images. IEEE Trans. on pattern analysis and machine intelligence (2007).

Digital Library

[23]

Si-Qi Liu, Xiangyuan Lan, and Pong C Yuen. 2018b. Remote photoplethysmography correspondence feature for 3d mask face presentation attack detection. In Proceedings of the European Conference on Computer Vision (ECCV). 558--573.

[24]

Yaojie Liu, Amin Jourabloo, and Xiaoming Liu. 2018a. Learning deep models for face anti-spoofing: Binary or auxiliary supervision. In Proceedings of the CVPR .

[25]

Jukka M"a"att"a, Abdenour Hadid, and Matti Pietik"ainen. 2011. Face spoofing detection from single images using micro-texture analysis. In 2011 international joint conference on Biometrics (IJCB). IEEE, 1--7.

[26]

David Menotti, Giovani Chiachia, Allan Pinto, William Robson Schwartz, Helio Pedrini, Alexandre Xavier Falcao, and Anderson Rocha. 2015. Deep representations for iris, face, and fingerprint spoofing detection. IEEE Transactions on Information Forensics and Security, Vol. 10, 4 (2015), 864--879.

Digital Library

[27]

Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the ICML .

[28]

Gang Pan, Shi Han, Zhaohui Wu, and Yueming Wang. 2005. 3D face recognition using mapped depth images. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)-Workshops. IEEE, 175--175.

[29]

Gang Pan, Lin Sun, Zhaohui Wu, and Shihong Lao. 2007. Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In Proceedings of the ICCV .

[30]

Aleksandr Parkin and Oleg Grinchuk. 2019. Recognizing Multi-Modal Face Spoofing With Face Recognition Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops . 0--0.

[31]

Keyurkumar Patel, Hu Han, and Anil K Jain. 2016. Cross-database face antispoofing with robust feature representation. In Proceedings of the CCBR .

[32]

Raghavendra Ramachandra and Christoph Busch. 2017. Presentation attack detection methods for face recognition systems: a comprehensive survey. ACM Computing Surveys (CSUR) (2017).

Digital Library

[33]

Ravi S Sandhu. 1998. Role-based access control. In Advances in computers . Vol. 46. Elsevier, 237--286.

[34]

Nitesh Saxena, Md Borhan Uddin, Jonathan Voris, and N Asokan. 2011. Vibrate-to-unlock: Mobile phone assisted user authentication to multiple personal RFID tags. In 2011 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 181--188.

[35]

Ajian Liu Chenxu Zhao Jun Wan* Sergio Escalera Hailin Shi Zezheng Wang Stan Z. Li Shifeng Zhang, Xiaobo Wang. 2019. A Dataset and Benchmark for Large-scale Multi-modal Face Anti-spoofing. In Proceedings of the CVPR .

[36]

Talha Ahmad Siddiqui, Samarth Bharadwaj, Tejas I Dhamecha, Akshay Agarwal, Mayank Vatsa, Richa Singh, and Nalini Ratha. 2016. Face anti-spoofing with multifeature videolet aggregation. In 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 1035--1040.

[37]

Xiaoyang Tan, Li Yi, Jun Liu, and Jiang Lin. 2010. Face Liveness Detection from a Single Image with Sparse Low Rank Bilinear Discriminative Model. In Proceedings of the ECCV .

[38]

Yan Wang, Fudong Nian, Teng Li, Zhijun Meng, and Kongqiao Wang. 2017. Robust face anti-spoofing with depth information. Journal of Visual Communication and Image Representation (2017).

[39]

Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A discriminative feature learning approach for deep face recognition. In Proceedings of the ECCV .

[40]

Sun-Hee Weon, Sung-Il Joo, and Hyung-Il Choi. 2014. Using Depth Information for Real-Time Face Detection. In International Conference on Human-Computer Interaction. Springer, 480--483.

[41]

Zhenqi Xu, Li Shan, and Weihong Deng. 2016. Learning temporal features using LSTM-CNN architecture for face anti-spoofing. In Pattern Recognition .

[42]

Jianwei Yang, Zhen Lei, and Stan Z Li. 2014. Learn convolutional neural network for face anti-spoofing. arXiv:1408.5601 (2014).

[43]

Jianwei Yang, Zhen Lei, Shengcai Liao, and Stan Z Li. 2013. Face liveness detection with component dependent descriptor. In 2013 International Conference on Biometrics (ICB). IEEE, 1--6.

[44]

Kim Youngshin, Na Jaekeun, Yoon Seongbeak, and Yi Juneho. 2009. Masked fake face detection using radiance measurements. J. Opt. Soc. Am. A (2009).

[45]

Zeng Zhihong, Pantic Maja, Glenn I Roisman, and Thomas S Huang. 2009. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence (2009).

Cited By

Antil ADhiman C(2025)Unmasking Deception: A Comprehensive Survey on the Evolution of Face Anti‐spoofing MethodsNeurocomputing10.1016/j.neucom.2024.128992617(128992)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.128992
Guo JLiu HLuo YHu XZou HZhang YLiu HZhao BCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Style-conditional Prompt Token Learning for Generalizable Face Anti-spoofingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680857(994-1003)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680857
He XLi YHe DYuan RHuang L(2024)Lightweight face anti-spoofing method based on cross-fused multi-modal featuresJournal of Electronic Imaging10.1117/1.JEI.33.2.02303333:02Online publication date: 1-Mar-2024
https://doi.org/10.1117/1.JEI.33.2.023033
Show More Cited By

Index Terms

Multi-modal Multi-layer Fusion Network with Average Binary Center Loss for Face Anti-spoofing
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Face anti-spoofing with multi-color double-stream CNN
ICDSC 2019: Proceedings of the 13th International Conference on Distributed Smart Cameras

Previous methods on face anti-spoofing rarely pay attention to the difference of multi-channel chrominance between genuine and fake faces, or only use hand crafted features, which cannot effectively fuse multi-channel chrominance information. This paper ...
Integration of image quality and motion cues for face anti-spoofing

A multi-cues integration framework is proposed using a hierarchical neural network.Bottleneck representations are effective in multi-cues feature fusion.Shearlet is utilized to perform face image quality assessment.Motion-based face liveness features ...
Multilevel fusing paired visible light and near-infrared spectral images for face anti-spoofing
Highlights
- Releasing a new dataset for face anti-spoofing.
- Proposing a novel method for ...
Abstract
Face anti-spoofing has become a vital element for guaranteeing the security of face recognition systems. Previous face anti-spoofing approaches generally exploit cues in visible light images or near-infrared images individually. Few ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
669
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)8

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Antil ADhiman C(2025)Unmasking Deception: A Comprehensive Survey on the Evolution of Face Anti‐spoofing MethodsNeurocomputing10.1016/j.neucom.2024.128992617(128992)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.128992
Guo JLiu HLuo YHu XZou HZhang YLiu HZhao BCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Style-conditional Prompt Token Learning for Generalizable Face Anti-spoofingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680857(994-1003)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680857
He XLi YHe DYuan RHuang L(2024)Lightweight face anti-spoofing method based on cross-fused multi-modal featuresJournal of Electronic Imaging10.1117/1.JEI.33.2.02303333:02Online publication date: 1-Mar-2024
https://doi.org/10.1117/1.JEI.33.2.023033
Wu JYin QSheng ZLu WHuang JLi B(2024)Audio Multi-View Spoofing Detection Framework Based on Audio-Text-Emotion CorrelationsIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.343188819(7133-7146)Online publication date: 2024
https://doi.org/10.1109/TIFS.2024.3431888
Kong CZheng KLiu YWang SRocha ALi H(2024)M3FAS: An Accurate and Robust MultiModal Mobile Face Anti-Spoofing SystemIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2024.338159821:6(5650-5666)Online publication date: Nov-2024
https://doi.org/10.1109/TDSC.2024.3381598
Liu WPan Y(2024)Spatio-Temporal-Based Action Face Anti-Spoofing Detection via Fusing Dynamics and Texture Face Keypoints CuesIEEE Transactions on Consumer Electronics10.1109/TCE.2024.336148070:1(2401-2413)Online publication date: Feb-2024
https://doi.org/10.1109/TCE.2024.3361480
Zhang LZheng DYuan MHan FWu ZLiu MLi X(2023)MultiSense: Cross-labelling and Learning Human Activities Using Multimodal Sensing DataACM Transactions on Sensor Networks10.1145/357826719:3(1-26)Online publication date: 17-Apr-2023
https://dl.acm.org/doi/10.1145/3578267
He DHe XXiang HYuan RNiu Y(2023)Wavelet transform-based two-stream convolutional networks for face anti-spoofingJournal of Electronic Imaging10.1117/1.JEI.32.1.01301532:01Online publication date: 1-Jan-2023
https://doi.org/10.1117/1.JEI.32.1.013015
Zhang DMeng JZhang JDeng XDing SZhou MWang QLi QChen Y(2023)SonarGuard: Ultrasonic Face Liveness Detection on Mobile DevicesIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.323630333:8(4401-4414)Online publication date: Aug-2023
https://doi.org/10.1109/TCSVT.2023.3236303
Zhang YZimmermann RYu ZGuo B(2023)Towards an Enhanced and Lightweight Face Authentication SystemCutting Edge Applications of Computational Intelligence Tools and Techniques10.1007/978-3-031-44127-1_10(211-228)Online publication date: 1-Dec-2023
https://doi.org/10.1007/978-3-031-44127-1_10
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten