skip to main content
10.1145/2660267.2660288acmconferencesArticle/Chapter ViewAbstractPublication PagesccsConference Proceedingsconference-collections
research-article

Blind Recognition of Touched Keys on Mobile Devices

Published: 03 November 2014 Publication History

Abstract

In this paper, we introduce a novel computer vision based attack that automatically discloses inputs on a touch-enabled device while the attacker cannot see any text or popup in a video of the victim tapping on the touch screen. We carefully analyze the shadow formation around the fingertip, apply the optical flow, deformable part-based model (DPM), k-means clustering and other computer vision techniques to automatically locate the touched points. Planar homography is then applied to map the estimated touched points to a reference image of software keyboard keys. Recognition of passwords is extremely challenging given that no language model can be applied to correct estimated touched keys. Our threat model is that a webcam, smartphone or Google Glass is used for stealthy attack in scenarios such as conferences and similar gathering places. We address both cases of tapping with one finger and tapping with multiple fingers and two hands. Extensive experiments were performed to demonstrate the impact of this attack. The per-character (or per-digit) success rate is over 97% while the success rate of recognizing 4-character passcodes is more than 90%. Our work is the first to automatically and blindly recognize random passwords (or passcodes) typed on the touch screen of mobile devices with a very high success rate.

References

[1]
M. Backes, T. Chen, M. Duermuth, H. Lensch, and M. Welk. Tempest in a teapot: Compromising reflections revisited. In Proceedings of 30th IEEE Symposium on Security and Privacy, pages 315--327, 2009.
[2]
M. Backes, M. Dürmuth, and D. Unruh. Compromising reflections or how to read lcd monitors around the corner. In Proceedings of IEEE Symposium on Security and Privacy, pages 158--169, 2008.
[3]
D. Balzarotti, M. Cova, and G. Vigna. Clearshot: Eavesdropping on keyboard input from video. In Proceedings of the 2008 IEEE Symposium on Security and Privacy, SP'08, pages 170--183, 2008.
[4]
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-up robust features (surf). Comput. Vis. Image Underst., 110(3):346--359, June 2008.
[5]
H. Benko, A. D. Wilson, and P. Baudisch. Precise selection techniques for multi-touch screens. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '06, pages 1263--1272, 2006.
[6]
R. Biddle, S. Chiasson, and P. van Oorschot. Graphical passwords: Learning from the first twelve years. In ACM Computing Surveys, 2012.
[7]
G. R. Bradski and A. Kaehler. Learning opencv, 1st edition. O'Reilly Media, Inc., first edition, 2008.
[8]
A. Bulling, F. Alt, and A. Schmidt. Increasing the security of gaze-based cued-recall graphical passwords using saliency\ masks. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2012.
[9]
J. Canny. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell., 8(6):679--698, 1986.
[10]
N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1 - Volume 01, CVPR '05,pages 886--893. IEEE Computer Society, 2005.
[11]
P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained, part-based models. IEEE Trans. Pattern Anal. Mach. Intell., 32:1627--1645, 2010.
[12]
C. Forlines, D. Wigdor, C. Shen, and R. Balakrishnan. Direct-touch vs. mouse input for tabletop displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '07, pages 647--656, 2007.
[13]
H. Grabner, M. Grabner, and H. Bischof. Real-time tracking via on-line boosting. In Proceedings of the British Machine Vision Conference, 2006.
[14]
R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2 edition, 2003.
[15]
S. B. Hirsch. Secure input system. In United States Patent No. 4,479,112, 1982.
[16]
S. B. Hirsch. Secure keyboard input terminal. In United States Patent No. 4,333,090, 1982.
[17]
B. Hoanca and K. Mock. Screen oriented technique for reducing the incidence of shoulder surfing. In Proceedings of the International Conference on Security and Management (SAM), 2005.
[18]
P. Huber. Robust Statistics. John Wiley & Sons, 1981.
[19]
Juniper Networks, Inc. Juniper networks third annual mobile threats report. http://www.juniper.net/us/en/local/pdf/additional-resources/3rd-jnpr-mobile-threats-report-exec-summary.pdf, 2013.
[20]
Y. Ke, R. Sukthankar, and M. Hebert. Efficient visual event detection using volumetric features. In Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 - Volume 01, ICCV '05, pages 166--173, 2005.
[21]
D. Kim, P. Dunphy, P. Briggs, J. Hook, J. W. Nicholson, J. Nicholson, and P. Olivier. Multi-touch authentication on tabletops. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2010.
[22]
I. Kim. Keypad against brute force attacks on smartphones. In IET Information Security, 2012.
[23]
J. Koch. Codescrambler. http://cydia.saurik.com/package/org.thebigboss.codescrambler/, 2014.
[24]
T. Lan, Y. Wang, and G. Mori. Discriminative figure-centric models for joint action localization and recognition. In\ International Conference on Computer Vision (ICCV), 2011.
[25]
C. Lee. System and method for secure data entry. In United States Patent Application Publication, 2011.
[26]
Logitech. Logitech hd pro webcam c920. http://www. logitech.com/en-us/product/hd-pro-webcam-c920, 2013.
[27]
D. G. Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision, 60(2):91--110, Nov. 2004.
[28]
F. Maggi, S. Gasparini, and G. Boracchi. A fast eavesdropping attack against touchscreens. In IAS, pages 320--325. IEEE, 2011.
[29]
J. Matas, C. Galambos, and J. Kittler. Robust detection of lines using the progressive probabilistic hough transform. Comput. Vis. Image Underst., 78(1):119--137, 2000.
[30]
K. E. McIntyre, J. F. Sheets, D. A. J. Gougeon, C. W.\ Watson, K. P. Morlang, and D. Faoro. Method for secure pin entry on touch screen display. In United States Patent No. 6,549,194, 2003.
[31]
M. Muja and D. G. Lowe. Fast approximate nearest neighbors with automatic algorithm configuration. In In VISAPP International Conference on Computer Vision Theory and Applications, pages 331--340, 2009.
[32]
Plugable. Plugable usb 2.0 digital microscope for windows, mac, linux (2mp, 10x-50x optical zoom, 200x digital magnification). http://www.amazon.com/Plugable-Digital-Microscope-Windows-Magnification/dp/B00AFH3IN4/ref=sr_1_1?ie=UTF8&qid=1382796731&sr=8--1&keywords=optical+zoom+ webcam, 2013.
[33]
R. Raguram, A. M. White, D. Goswami, F. Monrose, and J.-M. Frahm. ispy: automatic reconstruction of typed input from compromising reflections. In Proceedings of the 18th ACM conference on Computer and communications security, CCS '11, pages 527--536, 2011.
[34]
N. Sae-Bae, K. Ahmed, K. Isbister, and N. Memon. Biometric-rich gestures: A novel approach to authentication on multi-touch devices. In Proceedings of the 30th ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2012.
[35]
J. Shi and C. Tomasi. Good features to track. Technical,report, 1993.
[36]
H.-S. Shin. Device and method for inputting password using random keypad. In United States Patent No. 7,698,563, 2010.
[37]
X. Suo, Y. Zhu, and G. S. Owen. Graphical passwords: A survey. In Proceedings of Annual Computer Security Applications Conference (ACSAC), 2005.
[38]
R. Szeliski. Computer Vision: Algorithms and Applications. Springer-Verlag New York, Inc., 1st edition, 2010.
[39]
Y. Tian, R. Sukthankar, and M. Shah. Spatiotemporal deformable part models for action detection. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '13, pages 2642--2649, 2013.
[40]
F. Wang, X. Cao, X. Ren, and P. Irani. Detecting and leveraging finger orientation for interaction with direct-touch surfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology, UIST '09, pagesm 23--32, 2009.
[41]
F. Wang and X. Ren. Empirical evaluation for finger input properties in multi-touch interaction. In Proceedings of them SIGCHI Conference on Human Factors in Computing Systems, CHI '09, pages 1063--1072, 2009.
[42]
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. pages 3485--3492. IEEE, 2010.
[43]
Y. Xu, J. Heinly, A. M. White, F. Monrose, and J.-M. Frahm. Seeing double: Reconstructing obscured typed input from repeated compromising reflections. In Proceedings of the 20th ACM Conference on Computer and Communications Security (CCS), 2013.
[44]
Q. Yan, J. Han, Y. Li, J. Zhou, and R. H. Deng. Designing leakage-resilient password entry on touchscreen mobile mdevices. In Proceedings of the 8th ACM Symposium on Information, Computer and Communications Security (AsiaCCS), 2013.
[45]
Y.Hu, L. Cao, F.Lv, S.Yan, Y.Gong, and T.S.Huang. Action detection in complex scenes with spatial and temporal ambiguities. ICCV, 2009.
[46]
J. yves Bouguet. Pyramidal implementation of the lucas kanade feature tracker. Intel Corporation, Microprocessor Research Labs, 2000.
[47]
Y. Zhang, P. Xia, J. Luo, Z. Ling, B. Liu, and X. Fu. Fingerprint attack against touch-enabled devices. In Proceedings of the second ACM workshop on Security and privacy in smartphones and mobile devices, SPSM '12, pages 57--68, 2012.

Cited By

View all
  • (2025)Echoes of Fingertip: Unveiling POS Terminal Passwords Through Wi-Fi Beamforming FeedbackIEEE Transactions on Mobile Computing10.1109/TMC.2024.346556424:2(662-676)Online publication date: Feb-2025
  • (2024)Can virtual reality protect users from keystroke inference attacks?Proceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699053(2725-2742)Online publication date: 14-Aug-2024
  • (2024)ArmSpy++: Enhanced PIN Inference through Video-based Fine-grained Arm Posture AnalysisACM Transactions on Privacy and Security10.1145/3696418Online publication date: 23-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CCS '14: Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security
November 2014
1592 pages
ISBN:9781450329576
DOI:10.1145/2660267
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. computer vision attack
  2. mobile devices
  3. privacy enhancing keyboard

Qualifiers

  • Research-article

Funding Sources

Conference

CCS'14
Sponsor:

Acceptance Rates

CCS '14 Paper Acceptance Rate 114 of 585 submissions, 19%;
Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

Upcoming Conference

CCS '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)36
  • Downloads (Last 6 weeks)3
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Echoes of Fingertip: Unveiling POS Terminal Passwords Through Wi-Fi Beamforming FeedbackIEEE Transactions on Mobile Computing10.1109/TMC.2024.346556424:2(662-676)Online publication date: Feb-2025
  • (2024)Can virtual reality protect users from keystroke inference attacks?Proceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699053(2725-2742)Online publication date: 14-Aug-2024
  • (2024)ArmSpy++: Enhanced PIN Inference through Video-based Fine-grained Arm Posture AnalysisACM Transactions on Privacy and Security10.1145/3696418Online publication date: 23-Sep-2024
  • (2024)Pivot: Panoramic-Image-Based VR User Authentication against Side-Channel AttacksACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369497521:2(1-19)Online publication date: 9-Sep-2024
  • (2024)RefleXnoop: Passwords Snooping on NLoS Laptops Leveraging Screen-Induced Sound ReflectionProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3670341(3361-3375)Online publication date: 2-Dec-2024
  • (2024)MuKI-Fi: Multi-Person Keystroke Inference With BFI-Enabled Wi-Fi SensingIEEE Transactions on Mobile Computing10.1109/TMC.2024.336833923:10(9835-9850)Online publication date: Oct-2024
  • (2024)Video-Based Cryptanalysis: Extracting Cryptographic Keys from Video Footage of a Device’s Power LED Captured by Standard Video Cameras2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00163(2422-2440)Online publication date: 19-May-2024
  • (2024)Silent Thief: Password Eavesdropping Leveraging Wi-Fi Beamforming Feedback from POS TerminalIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621321(321-330)Online publication date: 20-May-2024
  • (2024)A Systematic Deconstruction of Human-Centric Privacy & Security Threats on Mobile PhonesInternational Journal of Human–Computer Interaction10.1080/10447318.2024.236151941:2(1628-1651)Online publication date: 12-Jun-2024
  • (2023)Hidden realityProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620286(859-876)Online publication date: 9-Aug-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media