research-article

GazeRevealer: Inferring Password Using Smartphone Front Camera

Authors:

Ibrahim Khalil,

Xianghua XuAuthors Info & Claims

MobiQuitous '18: Proceedings of the 15th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services

Pages 254 - 263

https://doi.org/10.1145/3286978.3287026

Published: 05 November 2018 Publication History

Abstract

The widespread use of smartphones has brought great convenience to our daily lives, while at the same time we have been increasingly exposed to security threats. Keystroke security is an essential element in user privacy protection. In this paper, we present GazeRevealer, a novel side-channel based keystroke inference framework to infer sensitive inputs on smartphone from video recordings of victim's eye patterns captured from smartphone front camera. We observe that eye movements typically follow the keystrokes typing on the number-only soft keyboard during password input. By exploiting eye patterns, we are able to infer the passwords being entered. We propose a novel algorithm to extract sensitive eye pattern images from video streams, and classify different eye patterns with Support Vector Classification. We also propose a novel enhanced method to boost the inference accuracy. Compared with prior keystroke detection approaches, GazeRevealer does not require any external auxiliary devices, and it relies only on smartphone front camera. We evaluate the performance of GazeRevealer with three different types of smartphones, and the result shows that GazeRevealer achieves 77.43% detection accuracy for a single key number and 83.33% inference rate for the 6-digit password in the ideal case.

References

[1]

T. Ahonen, A. Hadid, and M. Pietikainen. 2006. Face description with local binary patterns: Application to face recognition. IEEE Trans. PAMI 28, 12 (2006), 2037--2041.

Digital Library

[2]

A. Al-Rahayfeh and M. Faezipour. 2013. Eye tracking and head movement detection: A state-of-art survey. IEEE JTEHM 1 (2013).

[3]

K. Ali, A. X. Liu, W. Wang, and M. Shahzad. 2015. Keystroke recognition using win signals. In Proc. 21st Int. Conf. MobiCom. 90--102.

Digital Library

[4]

D. Balzarotti, M. Cova, and G. Vigna. 2008. Clearshot: Eavesdropping on keyboard input from video. In IEEE Sympo. Security and Privacy. 170--183.

Digital Library

[5]

L. Cai and H. Chen. 2011. TouchLogger: Inferring keystrokes on touch screen from smartphone motion. In Proc. 6th USENIX Workshop HotSec.

Digital Library

[6]

A. Eshmawi andS. Nair. 2013. Smartphone applications security: Survey of new vectors and solutions. In IEEE Int. Conf. AICCSA. 1--4.

[7]

D. W. Hansen and Q. Ji. 2010. In the eye of the beholder: A survey of models for eyes and gaze. IEEE Trans. PAMI 32, 3 (2010), 478--500.

Digital Library

[8]

C. W. Hsu and C.J. Lin. 2002. A comparison of methods for multiclass support vector machines. IEEE Trans. NN 13, 2 (2002), 415--425.

Digital Library

[9]

Q. Huang, A. Veeraraghavan, and A. Sabharwal. 2015. TabletGaze: unconstrained appearance-based gaze estimation in mobile tablets. arXiv:1508.01244 (2015).

[10]

P. Koutras and P. Maragos. 2015. Estimation of eye gaze direction angles based on active appearance models. In IEEE Int. Conf. on ICIP. 2424--2428.

[11]

O. Kubovic. 2016. 8 years of Android: malware, malicious apps, and how to stay safe. https://www.welivesecurity.com/2016/09/23/malicious-android-apps/. Accessed Sep. 23, 2016.

[12]

M. Li, Y. Meng, J. Liu, H. Zhu, X. Liang, Y. Liu, and N. Ruan. 2016. When CSI meets public WiFi: Inferring your mobile phone password via WiFi signals. In Proc. of the 2016 ACM SIGSAC Conf. on CCS. 1068--1079.

Digital Library

[13]

A. Mayberry, P. Hu, B. Marlin, C. Salthouse, and D. Ganesan. 2014. iShadow: design of a wearable, real-time mobile gaze tracker. In Proc. of the 12th Annual Int. Conf. on MobiSys. 82--94.

Digital Library

[14]

S. Narain, A. Sanatinia, and G. Noubir. 2014. Single-stroke language-agnostic keylogging using stereo-microphones and domain specific machine learning. In Proc. of the 2014 ACM Conf. on Security and privacy in wireless & mobile networks. ACM, 201--212.

Digital Library

[15]

E. Owusu, J. Han, S. Das, A. Perrig, and J. Zhang. 2012. ACCessory: password inference using accelerometers on smartphones. In Proc. of the 12th Workshop on Mobile Computing Systems & Applications. ACM, 9.

Digital Library

[16]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[17]

R. Schlegel, K. Zhang, X. Y. Zhou, M. Intwala, A. Kapadia, and X. F. Wang. 2011. Soundcomber: A Stealthy and Context-Aware Sound Trojan for Smartphones. In Proc. of NDSS. 17--33.

[18]

T. Shibata, D. M. Kim, J.and Hoffman, and M. S. Banks. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. Journal of vision 11, 8 (2011).

[19]

L. Simon and R. Anderson. 2013. Pin skimmer: Inferring pins through the camera and microphone. In Proc. of the 3rd ACM workshop on Security and privacy in smartphones & mobile devices. ACM, 67--78.

Digital Library

[20]

I. Sluganovic, M. Roeschlin, K. B. Rasmussen, and I. Martinovic. 2016. Using Reflexive Eye Movements for Fast Challenge-Response Authentication. In Proc. of the 2016 ACM SIGSAC Conference on CCS. 1056--1067.

Digital Library

[21]

J. C. Sun, X. C. Jin, Y. M. Chen, J. X. Zhang, Y. C. Zhang, and R. Zhang. 2016. VISIBLE: Video-Assisted Keystroke Inference from Tablet Backside Motion. In Proc. of NDSS.

[22]

R. Valenti and T. Gevers. 2012. Accurate eye center location through invariant isocentric patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 9 (2012), 1785--1798.

Digital Library

[23]

P. I. Wilson and J. Fernandez. 2006. Facial feature detection using Haar classifiers. Journal of Computing Sciences in Colleges 21, 4 (2006), 127--133.

Digital Library

[24]

A. Wulf. 2011. Stealing Passwords is Easy in Native Mobile Apps Despite OAuth. https://welcome.totheinter.net/2011/01/12/stealing-passwords-is-easy-in-native-mobile-apps-despite-oauth/.

[25]

Z. Xu, K. Bai, andS. C. Zhu. 2012. Taplogger: Inferring user inputs on smartphone touchscreens using on-board motion sensors. In Proc. of the 5th ACM conference on Security and Privacy in Wireless and Mobile Networks. ACM, 113--124.

Digital Library

[26]

T. Zhu, Q. Ma, S. F. Zhang, and Y. H. Liu. 2014. Context-free attacks using keyboard acoustic emanations. In Proc. of the 2014 ACM SIGSAC Conference on Computer and Communications Security. ACM, 453--464.

Digital Library

[27]

L. Zhuang, F. Zhou, and J. D. Tygar. 2009. Keyboard acoustic emanations revisited. ACM Trans. on Information and System Security (TISSEC) 13, 1 (2009), 3.

Digital Library

Cited By

Qiao HWang KHuang TXu XLiu SChen J(2024)A TEMPEST Attack Implementation based on Hidden Markov model in Smart GridJournal of Physics: Conference Series10.1088/1742-6596/2774/1/0120092774:1(012009)Online publication date: 1-Jul-2024
https://doi.org/10.1088/1742-6596/2774/1/012009
Feng DWang HChen S(2021)Wi-PW: Inferring Smartphone Password Using Wi-Fi SignalsProceedings of the 6th International Conference on Big Data and Computing10.1145/3469968.3469993(148-153)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1145/3469968.3469993
Jin WMurali SZhu HLi MKim YKim JVigna GShi E(2021)Periscope: A Keystroke Inference Attack Using Human Coupled Electromagnetic EmanationsProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security10.1145/3460120.3484549(700-714)Online publication date: 12-Nov-2021
https://dl.acm.org/doi/10.1145/3460120.3484549
Show More Cited By

Index Terms

GazeRevealer: Inferring Password Using Smartphone Front Camera
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
2. Security and privacy
  1. Human and societal aspects of security and privacy
    1. Privacy protections

Recommendations

Eye-gaze estimation under various head positions and iris states

A new method for eye-gaze estimation under normal head movement is investigated.Head position and orientations are acquired by Kinect depth data and eye direction is obtained from high resolution images.Bayesian multinomial logistic regression is used ...
ScreenGlint: Practical, In-situ Gaze Estimation on Smartphones
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

Gaze estimation has widespread applications. However, little work has explored gaze estimation on smartphones, even though they are fast becoming ubiquitous. This paper presents ScreenGlint, a novel approach which exploits the glint (reflection) of the ...
Head Pose-Invariant Eyelid and Iris Tracking Method

These days, there is more demand for a camera-based gaze estimation method for new interfaces and new marketing measurement tools. Considering these applications, the system should track a new user without any operation such as calibration. It should ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MobiQuitous '18: Proceedings of the 15th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services

November 2018

490 pages

ISBN:9781450360937

DOI:10.1145/3286978

General Chairs:
Henning Schulzrinne
Columbia University, USA
,
Pan Li
Case Western Reserve University, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

EAI: The European Alliance for Innovation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MobiQuitous '18

MobiQuitous '18: Computing, Networking and Services

November 5 - 7, 2018

NY, New York, USA

Acceptance Rates

Overall Acceptance Rate 26 of 87 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
132
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)8

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Qiao HWang KHuang TXu XLiu SChen J(2024)A TEMPEST Attack Implementation based on Hidden Markov model in Smart GridJournal of Physics: Conference Series10.1088/1742-6596/2774/1/0120092774:1(012009)Online publication date: 1-Jul-2024
https://doi.org/10.1088/1742-6596/2774/1/012009
Feng DWang HChen S(2021)Wi-PW: Inferring Smartphone Password Using Wi-Fi SignalsProceedings of the 6th International Conference on Big Data and Computing10.1145/3469968.3469993(148-153)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1145/3469968.3469993
Jin WMurali SZhu HLi MKim YKim JVigna GShi E(2021)Periscope: A Keystroke Inference Attack Using Human Coupled Electromagnetic EmanationsProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security10.1145/3460120.3484549(700-714)Online publication date: 12-Nov-2021
https://dl.acm.org/doi/10.1145/3460120.3484549
Kim HJoe BLiu Y(2020)TapSnoop: Leveraging Tap Sounds to Infer Tapstrokes on Touchscreen DevicesIEEE Access10.1109/ACCESS.2020.29662638(14737-14748)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2966263

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten