skip to main content
research-article

Integrating Handcrafted Features with Deep Representations for Smartphone Authentication

Published: 29 March 2022 Publication History

Abstract

Recent research demonstrates the potential of touch dynamics as a usable and privacy-preserving scheme for smartphone authentication. Most existing approaches rely on handcrafted features since deep models may be vulnerable to behavioral uncertainty due to the lack of consistent semantic information. Toward this end, we propose an approach to integrating handcrafted features into two phases of the deep learning process. On one hand, we present three fine-grained behavior representations by encoding semantic handcrafted features into the raw touch actions. On the other hand, we devise a deep Feature Regularization Net (FRN) architecture to combine the complementary information in both handcrafted and deep features. FRN involves handcrafted features as regularization to guide the learning process of deep features and selectively fuses these two feature types through a feature re-weighting mechanism. Experimental findings demonstrate that FRN outperforms the existing handcrafted or deep features even with smaller training and template sets. The framework also works for SOTA deep models and further boosts the accuracy. Results show that our approach is more reliable to alleviate behavioral variability and is competitively robust to statistical attacks compared with the most effective handcrafted features, suggesting a promising mechanism to improve the effectiveness and usability of behavioral authentication for multi-touch enabled mobile devices.

References

[1]
Mohammed Abuhamad, Ahmed Abusnaina, DaeHun Nyang, and David Mohaisen. 2020. Sensor-based Continuous Authentication of Smartphones' Users Using Behavioral Biometrics: A Contemporary Survey. IEEE Internet of Things Journal 8, 1 (2020), 65--84.
[2]
Alejandro Acien, John V Monaco, Aythami Morales, Ruben Vera-Rodriguez, and Julian Fierrez. 2020. Typenet: Scaling up keystroke biometrics. arXiv preprint arXiv:2004.03627 (2020).
[3]
Margit Antal and László Zsolt Szabó. 2016. Biometric authentication based on touchscreen swipe patterns. Procedia Technology 22 (2016), 862--869.
[4]
Adam J Aviv, Katherine L Gibson, Evan Mossop, Matt Blaze, and Jonathan M Smith. 2010. Smudge attacks on smartphone touch screens. Woot 10 (2010), 1--7.
[5]
Philip Bontrager, Aditi Roy, Julian Togelius, Nasir Memon, and Arun Ross. 2018. Deepmasterprints: Generating masterprints for dictionary attacks via latent variable evolution. In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS). IEEE, 1--9.
[6]
Léon Bottou. 2012. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade. Springer, 421--436.
[7]
Zhongmin Cai, Chao Shen, Miao Wang, Yunpeng Song, and Jialin Wang. 2013. Mobile authentication through touch-behavior features. In Chinese Conference on Biometric Recognition. Springer, 386--393.
[8]
Changhao Chen, Stefano Rosa, Yishu Miao, Chris Xiaoxuan Lu, Wei Wu, Andrew Markham, and Niki Trigoni. 2019. Selective sensor fusion for neural visual-inertial odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10542--10551.
[9]
Yimin Chen, Jingchao Sun, Rui Zhang, and Yanchao Zhang. 2015. Your song your way: Rhythm-based two-factor authentication for multi-touch mobile devices. In 2015 IEEE conference on computer communications (INFOCOM). IEEE, 2686--2694.
[10]
Zhenghua Chen, Chaoyang Jiang, Shili Xiang, Jie Ding, Min Wu, and Xiaoli Li. 2019. Smartphone sensor-based human activity recognition using feature fusion and maximum full a posteriori. IEEE Transactions on Instrumentation and Measurement 69, 7 (2019), 3992--4001.
[11]
Penny Chong, Yuval Elovici, and Alexander Binder. 2019. User authentication based on mouse dynamics using deep neural networks: A comprehensive study. IEEE Transactions on Information Forensics and Security 15 (2019), 1086--1101.
[12]
Alexander De Luca, Alina Hang, Frederik Brudy, Christian Lindner, and Heinrich Hussmann. 2012. Touch me once and i know it's you! implicit authentication based on touch screen patterns. In proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 987--996.
[13]
Google Developers. [n.d.]. Android API reference. https://developer.android.com/reference
[14]
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. 2021. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13733--13742.
[15]
Alaa El Masri, Harry Wechsler, Peter Likarish, Christopher Grayson, Calton Pu, Dalal Al-Arayed, and Brent ByungHoon Kang. 2015. Active authentication using scrolling behaviors. In 2015 6th International Conference on Information and Communication Systems (ICICS). IEEE, 257--262.
[16]
Julian Fierrez, Ada Pozo, Marcos Martinez-Diaz, Javier Galbally, and Aythami Morales. 2018. Benchmarking touchscreen biometrics for mobile authentication. IEEE transactions on information forensics and security 13, 11 (2018), 2720--2733.
[17]
Mario Frank, Ralf Biedert, Eugene Ma, Ivan Martinovic, and Dawn Song. 2012. Touchalytics: On the applicability of touchscreen input as a behavioral biometric for continuous authentication. IEEE transactions on information forensics and security 8, 1 (2012), 136--148.
[18]
Matteo Gadaleta and Michele Rossi. 2018. Idnet: Smartphone-based gait recognition with convolutional neural networks. Pattern Recognition 74 (2018), 25--37.
[19]
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks 18, 5-6 (2005), 602--610.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[21]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
[22]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).
[23]
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078 (2015).
[24]
Hassan Khan, Urs Hengartner, and Daniel Vogel. 2018. Evaluating attack and defense strategies for smartphone pin shoulder surfing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--10.
[25]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[26]
Pavel Korshunov and Sébastien Marcel. 2018. Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685 (2018).
[27]
Songxuan Lai, Lianwen Jin, Yecheng Zhu, Zhe Li, and Luojun Lin. 2021. SynSig2Vec: Forgery-free Learning of Dynamic Signature Representations by Sigma Lognormal-based Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[28]
Gaël Le Lan and Vincent Frey. 2019. Securing smartphone handwritten pin codes with recurrent neural networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2612--2616.
[29]
Yantao Li, Peng Tao, Shaojiang Deng, and Gang Zhou. 2021. DeFFusion: CNN-based Continuous Authentication Using Deep Feature Fusion. ACM Transactions on Sensor Networks (TOSN) 18, 2 (2021), 1--20.
[30]
Chenhao Lin and Ajay Kumar. 2018. A CNN-based framework for comparison of contactless to contact-based fingerprints. IEEE Transactions on Information Forensics and Security 14, 3 (2018), 662--676.
[31]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021).
[32]
Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. 2019. Bag of tricks and a strong baseline for deep person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 0--0.
[33]
Hao Luo, Wei Jiang, Youzhi Gu, Fuxu Liu, Xingyu Liao, Shenqi Lai, and Jianyang Gu. 2019. A strong baseline and batch normalization neck for deep person re-identification. IEEE Transactions on Multimedia 22, 10 (2019), 2597--2609.
[34]
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, Nov (2008), 2579--2605.
[35]
Natalia Neverova, Christian Wolf, Griffin Lacey, Lex Fridman, Deepak Chandra, Brandon Barbello, and Graham Taylor. 2016. Learning human identity from motion patterns. IEEE Access 4 (2016), 1810--1820.
[36]
Wei Niu, Xiaolong Ma, Yanzhi Wang, and Bin Ren. 2019. 26ms inference time for resnet-50: Towards real-time execution of all dnns on smartphone. arXiv preprint arXiv:1905.00571 (2019).
[37]
Shih Yin Ooi and Andrew Beng-Jin Teoh. 2019. Touch-stroke dynamics authentication using temporal regression forest. IEEE Signal Processing Letters 26, 7 (2019), 1001--1005.
[38]
Youcef Ouadjer, Mourad Adnane, and Nesrine Bouadjenek. 2021. Feature Importance Evaluation of Smartphone Touch Gestures for Biometric Authentication. In 2020 2nd International Workshop on Human-Centric Smart Environments for Health and Well-being (IHSH). IEEE, 103--107.
[39]
Napa Sae-Bae, Kowsar Ahmed, Katherine Isbister, and Nasir Memon. 2012. Biometric-rich gestures: a novel approach to authentication on multi-touch devices. In proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 977--986.
[40]
Napa Sae-Bae, Nasir Memon, Katherine Isbister, and Kowsar Ahmed. 2014. Multitouch gesture-based authentication. IEEE transactions on information forensics and security 9, 4 (2014), 568--582.
[41]
Abdul Serwadda and Vir V Phoha. 2013. Examining a large keystroke biometrics dataset for statistical-attack openings. ACM Transactions on Information and System Security (TISSEC) 16, 2 (2013), 1--30.
[42]
Abdul Serwadda and Vir V Phoha. 2013. When kids' toys breach mobile phone security. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security. 599--610.
[43]
Muhammad Shahzad, Alex X Liu, and Arjmand Samuel. 2013. Secure unlocking of mobile touch screen devices by simple gestures: you can see it but you can not do it. In Proceedings of the 19th annual international conference on Mobile computing & networking. 39--50.
[44]
Richard Shay, Saranga Komanduri, Adam L Durity, Phillip Huh, Michelle L Mazurek, Sean M Segreti, Blase Ur, Lujo Bauer, Nicolas Christin, and Lorrie Faith Cranor. 2016. Designing password policies for strength and usability. ACM Transactions on Information and System Security (TISSEC) 18, 4 (2016), 1--34.
[45]
Chao Shen, Yuanxun Li, Yufei Chen, Xiaohong Guan, and Roy A Maxion. 2017. Performance analysis of multi-motion sensor behavior for active smartphone authentication. IEEE Transactions on Information Forensics and Security 13, 1 (2017), 48--62.
[46]
Chao Shen, Yong Zhang, Xiaohong Guan, and Roy A Maxion. 2015. Performance analysis of touch-interaction behavior for active smartphone authentication. IEEE Transactions on Information Forensics and Security 11, 3 (2015), 498--513.
[47]
Michael Sherman, Gradeigh Clark, Yulong Yang, Shridatt Sugrim, Arttu Modig, Janne Lindqvist, Antti Oulasvirta, and Teemu Roos. 2014. User-generated free-form gestures for authentication: Security and memorability. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. 176--189.
[48]
Dai Shi, Dan Tao, Jiangtao Wang, Muyan Yao, Zhibo Wang, Houjin Chen, and Sumi Helal. 2021. Fine-Grained and Context-Aware Behavioral Biometrics for Pattern Lock on Smartphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1--30.
[49]
Zdeňka Sitová, Jaroslav Šeděnka, Qing Yang, Ge Peng, Gang Zhou, Paolo Gasti, and Kiran S Balagani. 2015. HMOG: New behavioral biometric features for continuous authentication of smartphone users. IEEE Transactions on Information Forensics and Security 11, 5 (2015), 877--892.
[50]
Yunpeng Song, Zhongmin Cai, and Zhi-Li Zhang. 2017. Multi-touch authentication using hand geometry and behavioral information. In 2017 IEEE symposium on security and privacy (SP). IEEE, 357--372.
[51]
Yunpeng Song, Cori Faklaris, Zhongmin Cai, Jason I Hong, and Laura Dabbish. 2019. Normal and Easy: Account Sharing Practices in the Workplace. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--25.
[52]
Valeriu-Daniel Stanciu, Riccardo Spolaor, Mauro Conti, and Cristiano Giuffrida. 2016. On the effectiveness of sensor-enhanced keystroke dynamics against statistical attacks. In proceedings of the sixth ACM conference on data and application security and privacy. 105--112.
[53]
Ruben Tolosana, Ruben Vera-Rodriguez, Julian Fierrez, and Javier Ortega-Garcia. 2020. BioTouchPass2: Touchscreen Password Biometrics Using Time-Aligned Recurrent Neural Networks. IEEE Transactions on Information Forensics and Security 15 (2020), 2616--2628.
[54]
TSYS. 2018. U.S. Consumer Payment Study. (2018). https://www.tsys.com/Assets/TSYS/downloads/rs_2018-us-consumer-payment-study.pdf
[55]
Verizon. 2020. 2020 Data Breach Investigations Report. https://enterprise.verizon.com/resources/reports/dbir/
[56]
Jiayun Wang, Patrick Virtue, and Stella X Yu. 2017. Successive Embedding and Classification Loss for Aerial Image Classification. arXiv preprint arXiv:1712.01511 (2017).
[57]
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. 2017. Deep metric learning with angular loss. In Proceedings of the IEEE International Conference on Computer Vision. 2593--2601.
[58]
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A discriminative feature learning approach for deep face recognition. In European conference on computer vision. Springer, 499--515.
[59]
Shangxuan Wu, Ying-Cong Chen, Xiang Li, An-Cong Wu, Jin-Jie You, and Wei-Shi Zheng. 2016. An enhanced deep feature representation for person re-identification. In 2016 IEEE winter conference on applications of computer vision (WACV). IEEE, 1--8.
[60]
Hui Xu, Yangfan Zhou, and Michael R Lyu. 2014. Towards continuous and passive authentication via touch biometrics: An experimental study on smartphones. In 10th Symposium On Usable Privacy and Security ({SOUPS} 2014). 187--198.
[61]
Xiangyu Xu, Jiadi Yu, Yingying Chen, Qin Hua, Yanmin Zhu, Yi-Chao Chen, and Minglu Li. 2020. TouchPass: towards behavior-irrelevant on-touch user authentication on smartphones leveraging vibrations. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--13.
[62]
Daksha Yadav, Naman Kohli, Akshay Agarwal, Mayank Vatsa, Richa Singh, and Afzel Noore. 2018. Fusion of handcrafted and deep learning features for large-scale multiple iris presentation attack detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 572--579.
[63]
Muhammad Rehman Zafar and Munam Ali Shah. 2016. Fingerprint authentication and security risks in smart devices. In 2016 22nd International Conference on Automation and Computing (ICAC). IEEE, 548--553.
[64]
Ahmad Zairi Zaidi, Chun Yong Chong, Zhe Jin, Rajendran Parthiban, and Ali Safaa Sadiq. 2021. Touch-based continuous mobile device authentication: State-of-the-art, challenges and opportunities. Journal of Network and Computer Applications (2021), 103162.
[65]
Heng Zhang, Vishal M Patel, Mohammed Fathy, and Rama Chellappa. 2015. Touch gesture-based active user authentication using dictionaries. In 2015 IEEE Winter Conference on Applications of Computer Vision. IEEE, 207--214.
[66]
Xinchen Zhang, Yafeng Yin, Lei Xie, Hao Zhang, Zefan Ge, and Sanglu Lu. 2020. TouchID: User Authentication on Mobile Devices via Inertial-Touch Gesture Analysis. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--29.
[67]
Xi Zhao, Tao Feng, and Weidong Shi. 2013. Continuous mobile authentication using a novel graphic touch gesture feature. In 2013 IEEE sixth international conference on biometrics: theory, applications and systems (BTAS). IEEE, 1--6.

Cited By

View all
  • (2025)ScooterID: Posture-Based Continuous User Identification From Mobility Scooter RidesIEEE Transactions on Mobile Computing10.1109/TMC.2024.347360924:2(970-984)Online publication date: 1-Feb-2025
  • (2024)CoreTemp: Coreset Sampled Templates for Multimodal Mobile BiometricsApplied Sciences10.3390/app1412518314:12(5183)Online publication date: 14-Jun-2024
  • (2024)Modeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information SeekingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997508:4(1-28)Online publication date: 21-Nov-2024
  • Show More Cited By

Index Terms

  1. Integrating Handcrafted Features with Deep Representations for Smartphone Authentication

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
      Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 6, Issue 1
      March 2022
      1009 pages
      EISSN:2474-9567
      DOI:10.1145/3529514
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 29 March 2022
      Published in IMWUT Volume 6, Issue 1

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Behavioral authentication
      2. Biometrics
      3. Feature fusion
      4. Mobile devices

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)88
      • Downloads (Last 6 weeks)8
      Reflects downloads up to 01 Mar 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)ScooterID: Posture-Based Continuous User Identification From Mobility Scooter RidesIEEE Transactions on Mobile Computing10.1109/TMC.2024.347360924:2(970-984)Online publication date: 1-Feb-2025
      • (2024)CoreTemp: Coreset Sampled Templates for Multimodal Mobile BiometricsApplied Sciences10.3390/app1412518314:12(5183)Online publication date: 14-Jun-2024
      • (2024)Modeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information SeekingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997508:4(1-28)Online publication date: 21-Nov-2024
      • (2024)Touch Authentication for Sharing Context Using Within-Group Similarity StructureIEEE Internet of Things Journal10.1109/JIOT.2024.340232311:17(28281-28296)Online publication date: 1-Sep-2024
      • (2024)Station: Gesture-Based Authentication for Voice InterfacesIEEE Internet of Things Journal10.1109/JIOT.2024.338272111:12(22668-22683)Online publication date: 15-Jun-2024
      • (2024)AttAuth: An Implicit Authentication Framework for Smartphone Users Using Multimodality DataIEEE Internet of Things Journal10.1109/JIOT.2023.331471711:4(6928-6942)Online publication date: 15-Feb-2024
      • (2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: 23-Jan-2024
      • (2023)Deep Learning and Machine Learning, Better Together Than Apart: A Review on Biometrics Mobile AuthenticationJournal of Cybersecurity and Privacy10.3390/jcp30200133:2(227-258)Online publication date: 13-Jun-2023
      • (2023)Integrating Gaze and Mouse Via Joint Cross-Attention Fusion Net for Students' Activity Recognition in E-learningProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108767:3(1-35)Online publication date: 27-Sep-2023
      • (2023)Genie in the ModelProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808157:1(1-29)Online publication date: 28-Mar-2023
      • Show More Cited By

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media