research-article

Pretrained Implicit-Ensemble Transformer for Open-Set Authentication on Multimodal Mobile Biometrics

Authors:

Kensuke Wagata,

Andrew Beng Jin TeohAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 5909 - 5922

https://doi.org/10.1145/3581783.3612467

Published: 27 October 2023 Publication History

Abstract

Smartphones have become indispensable in our lives, even for security-critical tasks. Traditional security measures such as PINs provide only one-time authentication, while biometrics enable continuous authentication in mobile devices. This paper introduces a simple, lightweight, pretrained Transformer dubbed PIEformer for open-set authentication (OSA) of multimodal touchstrokes and gait biometrics. Compared to conventional mobile closed-set authentication, OSA enables more secure and practical authentication, with genuine and impostor users disjoint from the training set. PIEFormer incorporates a novel implicit ensembling mechanism for extracting discriminative embeddings within an open-set environment and enhancing generalization performance. This approach learns multiple diverse sub-embeddings, capturing complementary aspects of biometrics data with minimal computational overhead, allowing Transformers to exhibit robust capabilities in OSA. Our proposed methods demonstrate state-of-the-art results on HMOG and BBMAS datasets, particularly in open-set scenarios compared to closed-set literature, thus bringing mobile biometric authentication closer to real-world applications.

References

[1]

Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928.

[2]

Mohit Agrawal, Pragyan Mehrotra, Rajesh Kumar, and Rajiv Ratn Shah. 2021. Defending touch-based continuous authentication systems from active adversaries using generative adversarial networks. In 2021 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 1--8.

Digital Library

[3]

Cezara Benegui and Radu Tudor Ionescu. 2020. To augment or not to augment? data augmentation in user identification based on motion sensors. In Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18-22, 2020, Proceedings, Part V 27. Springer, 822--831.

[4]

Cezara Benegui and Radu Tudor Ionescu. 2020. To augment or not to augment? data augmentation in user identification based on motion sensors. In Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18-22, 2020, Proceedings, Part V 27. Springer, 822--831.

[5]

Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. 2020. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI. Springer, 548--564.

[6]

Mario Parreno Centeno, Aad van Moorsel, and Stefano Castruccio. 2017. Smartphone continuous authentication using deep learning autoencoders. In 2017 15th annual conference on privacy, security and trust (pst). IEEE, 147--1478.

[7]

Hao Chen and Abhinav Shrivastava. 2020. Group ensemble: learning an ensemble of convnets in a single convnet. arXiv preprint arXiv:2007.00649.

[8]

Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). Vol. 1. IEEE, 539--546.

Digital Library

[9]

Shaojiang Deng, Jiaxing Luo, and Yantao Li. 2021. Cnn-based continuous authentication on smartphones with auto augmentation search. In Information and Communications Security: 23rd International Conference, ICICS 2021, Chongqing, China, November 19--21, 2021, Proceedings, Part I 23. Springer, 169--186.

Digital Library

[10]

Alexey Dosovitskiy et al. 2020. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

[11]

Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. 2018. Dropblock: a regularization method for convolutional networks. Advances in neural information processing systems, 31.

[12]

Giacomo Giorgi, Andrea Saracino, and Fabio Martinelli. 2021. Using recurrent neural networks for continuous authentication through gait analysis. Pattern Recognition Letters, 147, 157--163.

Digital Library

[13]

Xing Guan, Sihua Yang, Hao Zhang, and Jingkuan Song. 2021. Gait recognition with transformer. Multimedia Tools and Applications, 80, 22, 33633--33649.

[14]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778.

[15]

Andrew Howard et al. 2019. Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision, 1314--1324.

[16]

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

[17]

Hailong Hu, Yantao Li, Zhangqian Zhu, and Gang Zhou. 2018. Cnnauth: continuous authentication via two-stream convolutional neural networks. In 2018 IEEE international conference on networking, architecture and storage (NAS). IEEE, 1--9.

[18]

Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. 2017. Snapshot ensembles: train 1, get m for free. arXiv preprint arXiv:1704.00109.

[19]

Özlem Durmaz Incel, Seçil Günay, Yasemin Akan, Yunus Barlas, Okan Engin Basar, Gülfem Isiklar Alptekin, and Mustafa Isbilen. 2021. Dakota: sensor and touch screen-based continuous authentication on a mobile banking application. IEEE Access, 9, 38943--38960.

[20]

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with Gpu and Faiss. arXiv preprint arXiv:1702.08734.

[21]

Christy James Jose and MS Rajasree. 2022. Deep learning-based implicit continuous authentication of smartphone user. In Proceedings of Third International Conference on Communication, Computing and Electronics Systems: ICCCES 2021. Springer, 387--400.

[22]

Amith K. Belman, Li Wang, Sundaraja S. Iyengar, Pawel Sniatala, Robert Wright, Robert Dora, Jacob Baldwin, Zhanpeng Jin, and Vir V. Phoha. 2019. Su-ais bb-mas (syracuse university and assured information security - behavioral biometrics multi-device and multi-activity data from same users) dataset. (2019).

[23]

Mahmut Kaya and Hasan ?akir Bilge. 2019. Deep metric learning: a survey. Symmetry, 11, 9, 1066.

[24]

Yantao Li, Hailong Hu, Zhangqian Zhu, and Gang Zhou. 2020. Scanet: sensor-based continuous authentication with two-stream convolutional neural networks. ACM Transactions on Sensor Networks (TOSN), 16, 3, 1--27.

Digital Library

[25]

Yantao Li, Peng Tao, Shaojiang Deng, and Gang Zhou. 2021. Deffusion: cnnbased continuous authentication using deep feature fusion. ACM Transactions on Sensor Networks (TOSN), 18, 2, 1-20.

[26]

Sakorn Mekruksavanich, Ponnipa Jantawong, and Anuchit Jitpattanakul. 2022. Enhancement of sensor-based user identification using data augmentation techniques. In 2022 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON). IEEE, 333--337.

[27]

Sakorn Mekruksavanich and Anuchit Jitpattanakul. 2021. Deep learning approaches for continuous authentication based on activity patterns using mobile sensing. Sensors, 21, 22, 7519.

[28]

Sakorn Mekruksavanich and Anuchit Jitpattanakul. 2021. Deep learning approaches for continuous authentication based on activity patterns using mobile sensing. Sensors, 21, 22, 7519.

[29]

Andrew Ray-Dowling, Dongyan Hou, and Stephanie Schuckers. 2023. Stationary mobile behavioral biometrics: a survey. Computers & Security, (Mar. 2023), 103184.

Digital Library

[30]

Praveen Kumar Rayani and Suvamoy Changder. 2023. Continuous user authentication on smartphone via behavioral biometrics: a survey. Multimedia Tools and Applications, 82, 2, 1633--1667.

Digital Library

[31]

Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. 2018. Deep one-class classification. In International conference on machine learning. PMLR, 4393--4402.

[32]

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4510--4520.

[33]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: a unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 815--823.

[34]

Zdeňka Sitová, Jaroslav ?eděnka, Qing Yang, Ge Peng, Gang Zhou, Paolo Gasti, and Kiran S Balagani. 2015. Hmog: new behavioral biometric features for continuous authentication of smartphone users. IEEE Transactions on Information Forensics and Security, 11, 5, 877--892.

Digital Library

[35]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15, 1, 1929--1958.

[36]

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning. PMLR, 20827--20840.

[37]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

[38]

Hasan Can Volaka, Gulfem Alptekin, Okan Engin Basar, Mustafa Isbilen, and Ozlem Durmaz Incel. 2019. Towards continuous authentication on mobile phones using deep learning models. Procedia Computer Science, 155, 177--184.

[39]

Kensuke Wagata and Andrew Beng Jin Teoh. 2022. Few-shot continuous authentication for mobile-based biometrics. Applied Sciences, 12, 20, 10365.

[40]

Tianshu Wang, Yan Huang, Hongzhi Liu, and Jian Yang. 2021. Ktrans: a transformer-based keystroke dynamics authentication system. IEEE Access, 9, 68037--68048.

[41]

Xiaobo Wang, Hao Cheng, Xinyu Zhang, and Zheng Wang. 2021. Vision transformers for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 25840--25849.

[42]

Yeming Wen, Dustin Tran, and Jimmy Ba. 2020. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715.

[43]

Wei Wu, Qiong Wu, and Xiangfeng Chen. 2021. A novel authentication scheme for fingerprint based on transformer. IEEE Access, 9, 81362--81373.

[44]

Cheng Yang, Shuang Li, Yan Liu, and Ming Li. 2021. Transformer-based iris recognition. IEEE Access, 9, 111775--111781.

[45]

Wenzhao Zheng, Chengkun Wang, Jiwen Lu, and Jie Zhou. 2021. Deep compositional metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9320--9329.

[46]

Zhi-Hua Zhou. 2012. Ensemble methods: foundations and algorithms. CRC press.

Digital Library

Cited By

Zhang ZYin HRao SYan XWang ZLiang WZhao YShan YZhang RLin YJiang J(2025)Identifying E-Commerce Fraud Through User Behavior Data: Observations and InsightsData Science and Engineering10.1007/s41019-024-00275-610:1(24-39)Online publication date: 15-Jan-2025
https://doi.org/10.1007/s41019-024-00275-6
Yoon JPark JKim JTeoh A(2024)CoreTemp: Coreset Sampled Templates for Multimodal Mobile BiometricsApplied Sciences10.3390/app1412518314:12(5183)Online publication date: 14-Jun-2024
https://doi.org/10.3390/app14125183
Chen FXin JPhoha V(2024)SSPRA: A Robust Approach to Continuous Authentication Amidst Real-World Adversarial ChallengesIEEE Transactions on Biometrics, Behavior, and Identity Science10.1109/TBIOM.2024.33695906:2(245-260)Online publication date: Apr-2024
https://doi.org/10.1109/TBIOM.2024.3369590

Index Terms

Pretrained Implicit-Ensemble Transformer for Open-Set Authentication on Multimodal Mobile Biometrics
1. Security and privacy
  1. Intrusion/anomaly detection and malware mitigation
    1. Intrusion detection systems
      1. Artificial immune systems
  2. Security services
    1. Authentication
      1. Biometrics

Recommendations

Multi-factor biometrics for authentication: a false sense of security
MM&Sec '10: Proceedings of the 12th ACM workshop on Multimedia and security

Multi-factor biometric authentications have been proposed recently to strengthen security and/or privacy of biometric systems in addition to enhancing authentication accuracy. An important approach to multi-factor biometric authentication is to apply ...
Continuous Authentication Using Behavioral Biometrics

A continuous behaviometric authentication system is tested on 99 users over 10 weeks, focusing on keystroke dynamics, mouse movements, application usage, and the system footprint. In the process, a new trust model was created to enable continuous ...
A Concept of Continuous User Authentication Based on Behavioral Biometrics
FRUCT '17: Proceedings of the 20th Conference of Open Innovations Association FRUCT

This paper focuses on continuous user authentication based on its interaction with the device. Behavioral authentication provides the ability to partially abandon passwords. Furthermore, the use of human behavior, for example, how he holds the device in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP)

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
160
Total Downloads

Downloads (Last 12 months)77
Downloads (Last 6 weeks)5

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZYin HRao SYan XWang ZLiang WZhao YShan YZhang RLin YJiang J(2025)Identifying E-Commerce Fraud Through User Behavior Data: Observations and InsightsData Science and Engineering10.1007/s41019-024-00275-610:1(24-39)Online publication date: 15-Jan-2025
https://doi.org/10.1007/s41019-024-00275-6
Yoon JPark JKim JTeoh A(2024)CoreTemp: Coreset Sampled Templates for Multimodal Mobile BiometricsApplied Sciences10.3390/app1412518314:12(5183)Online publication date: 14-Jun-2024
https://doi.org/10.3390/app14125183
Chen FXin JPhoha V(2024)SSPRA: A Robust Approach to Continuous Authentication Amidst Real-World Adversarial ChallengesIEEE Transactions on Biometrics, Behavior, and Identity Science10.1109/TBIOM.2024.33695906:2(245-260)Online publication date: Apr-2024
https://doi.org/10.1109/TBIOM.2024.3369590

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten