Ülkü Meteriz-Yıldıran
ABSTRACT
The convergence of various technologies, such as smartwatches, smartphones, etc. has proven to be beneficial, although poses various security and privacy risks. In this paper, we explore one such risk where a smartwatch can be exploited to infer what a user is typing on a physical keyboard while wearing the smartwatch. We exploited the acoustic emanations of the keyboard as recorded by the smartwatch to perform the proposed attack-SIA. To address various environment-related challenges, SIA employs four stages: Noise Cancelling, Keystroke Detection, Key Identification, and Word Correction, where several digital signal processing, machine learning, and natural language processing techniques are utilized to produce the final inference. Our results show that an acoustic emanation of a physical keyboard captured by a smartwatch recovers up to 98% of the typed text. We also showed that utilizing the noise cancellation, SIA is robust to the changes in the attack environment, which further boosts the practicality of the attack. The findings are alarming and call for further investigation on methods to cope with inference attacks due to the convergence of those technologies.
Supplemental Material
- 2020. Smartwatch Market - Growth, Trends, Forecasts (2020 - 2025). https://www.researchandmarkets.com/reports/4591978/smartwatch-market- growth-trends-forecastGoogle Scholar
- Manal Al-Sharrah, Ayed Salman, and Imtiaz Ahmad. 2018. Watch Your Smartwatch. In 2018 International Conference on Computing Sciences and Engineering (ICCSE). 1--5. https://doi.org/10.1109/ICCSE1.2018.8374228Google Scholar
- Kamran Ali, Alex X. Liu, Wei Wang, and Muhammad Shahzad. 2015. Keystroke Recognition Using WiFi Signals. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (Paris, France) (MobiCom '15). Association for Computing Machinery, New York, NY, USA, 90--102. https://doi.org/10.1145/2789168.2790109Google ScholarDigital Library
- D. Asonov and R. Agrawal. 2004. Keyboard acoustic emanations. In IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004. 3--11. https://doi.org/10.1109/SECPRI.2004.1301311Google ScholarCross Ref
- Ibrahim Baggili, Jeff Oduro, Kyle Anthony, Frank Breitinger, and Glenn McGee. 2015. Watch What You Wear: Preliminary Forensic Analysis of Smart Watches. In 2015 10th International Conference on Availability, Reliability and Security. 303--311. https://doi.org/10.1109/ARES.2015.39Google ScholarDigital Library
- D. Balzarotti, M. Cova, and G. Vigna. 2008. ClearShot: Eavesdropping on Keyboard Input from Video. In 2008 IEEE Symposium on Security and Privacy (sp 2008). 170--183.Google ScholarDigital Library
- Bianco D. Barisani A. 2009. Sniffing Keystrokes With Lasers and Voltmeters. In In Proceedings of Black Hat USA.Google Scholar
- Vincent Becker, Linus Fessler, and Gábor Sörös. 2019. GestEar: Combining Audio and Motion Sensing for Gesture Recognition on Smartwatches. In Proceedings of the 23rd International Symposium on Wearable Computers (London, United Kingdom) (ISWC '19). Association for Computing Machinery, New York, NY, USA, 10--19. https://doi.org/10.1145/3341163.3347735Google ScholarDigital Library
- Yigael Berger, Avishai Wool, and Arie Yeredor. 2006. Dictionary Attacks Using Keyboard Acoustic Emanations. In Proceedings of the 13th ACM Conference on Computer and Communications Security (Alexandria, Virginia, USA) (CCS '06). Association for Computing Machinery, New York, NY, USA, 245--254. https://doi.org/10.1145/1180405.1180436Google ScholarDigital Library
- Bo Chen, Vivek Yenamandra, and Kannan Srinivasan. 2015. Tracking Keystrokes Using Wireless Signals. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (Florence, Italy) (MobiSys '15). Association for Computing Machinery, New York, NY, USA, 31--44. https://doi.org/10.1145/2742647.2742673Google ScholarDigital Library
- Alberto Compagno, Mauro Conti, Daniele Lain, and Gene Tsudik. 2017. Don't Skype & Type! Acoustic Eavesdropping in Voice-Over-IP. In Proceedings ACM on Asia Conference on Computer and Communications Security (Abu Dhabi, United Arab Emirates). ACM, New York, NY, USA, 703--715. https://doi.org/10.1145/3052973.3053005Google ScholarDigital Library
- Stanford Center for Digital Health and Rock Health. 2019. https://rockhealth.com/reports/digital-health-consumer-adoption-report-2019/Google Scholar
- Jeffrey Friedman. 1972. Tempest: A signal problem. NSA Cryptologic Spectrum, Vol. 35 (1972), 76.Google Scholar
- Jun Gong, Zheer Xu, Qifan Guo, Teddy Seyed, Xiang 'Anthony' Chen, Xiaojun Bi, and Xing-Dong Yang. 2018. WrisText: One-Handed Text Entry on Smartwatch Using Wrist Gestures .Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 1--14. https://doi.org/10.1145/3173574.3173755Google ScholarDigital Library
- M. Hagiwara. 2021. Real-World Natural Language Processing: Practical applications with deep learning. Manning Publications. https://books.google.com.tr/books?id=A92_zQEACAAJGoogle Scholar
- Tzipora Halevi and Nitesh Saxena. 2014. Keyboard acoustic side channel attacks: exploring realistic and security-sensitive scenarios. International Journal of Information Security, Vol. 14 (09 2014), 1--14. https://doi.org/10.1007/s10207-014-0264-7Google ScholarDigital Library
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR, Vol. abs/1412.6980 (2014). arxiv: 1412.6980 http://arxiv.org/abs/1412.6980Google Scholar
- Bryan Klimt and Yiming Yang. 2004. The Enron Corpus: A New Dataset for Email Classification Research. In Proceedings of the 15th European Conference on Machine Learning (Pisa, Italy) (ECML'04). Springer-Verlag, Berlin, Heidelberg, 217--226. https://doi.org/10.1007/978-3-540-30115-8_22Google ScholarDigital Library
- Gierad Laput and Chris Harrison. 2019. Sensing Fine-Grained Hand Activity with Smartwatches. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3290605.3300568Google ScholarDigital Library
- Ho-Man Colman Leung, Chi-Wing Fu, and Pheng-Ann Heng. 2018. TwistIn: Tangible Authentication of Smart Devices via Motion Co-Analysis with a Smartwatch. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, 2, Article 72 (July 2018), 24 pages. https://doi.org/10.1145/3214275Google ScholarDigital Library
- Vladimir I Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, Vol. 10 (February 1966), 707.Google Scholar
- S. Li, A. Ashok , Y. Zhang, C. Xu, J. Lindqvist, and M. Gruteser. 2016. Whose move is it anyway? Authenticating smart wearable devices using unique head movement patterns. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom). 1--9. https://doi.org/10.1109/PERCOM.2016.7456514Google ScholarCross Ref
- Xiangyu Liu, Zhe Zhou, Wenrui Diao, Zhou Li, and Kehuan Zhang. 2015. When Good Becomes Evil: Keystroke Inference with Smartwatch. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (Denver, Colorado, USA) (CCS '15). Association for Computing Machinery, New York, NY, USA, 1273--1285. https://doi.org/10.1145/2810103.2813668Google ScholarDigital Library
- Chris Xiaoxuan Lu, Bowen Du, Peijun Zhao, Hongkai Wen, Yiran Shen, Andrew Markham, and Niki Trigoni. 2018. Deepauth: In-Situ Authentication for Smartwatches via Deeply Learned Behavioural Biometrics. In Proceedings of the 2018 ACM International Symposium on Wearable Computers (Singapore, Singapore) (ISWC '18). Proceedings of the 2018 ACM International Symposium on Wearable Computers, New York, NY, USA, 204--207. https://doi.org/10.1145/3267242.3267252Google ScholarDigital Library
- Anindya Maiti, Oscar Armbruster, Murtuza Jadliwala, and Jibo He. 2016. Smartwatch-Based Keystroke Inference Attacks and Context-Aware Protection Mechanisms. In Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security (China) (ASIA CCS '16). Association for Computing Machinery, New York, NY, USA, 795--806. https://doi.org/10.1145/2897845.2897905Google ScholarDigital Library
- Philip Marquardt, Arunabh Verma, Henry Carter, and Patrick Traynor. 2011. (Sp)IPhone: Decoding Vibrations from Nearby Keyboards Using Mobile Phone Accelerometers. In 18th ACM Conference on Computer and Communications Security (Chicago, Illinois, USA) (CCS '11). ACM, New York, NY, USA, 551--562. https://doi.org/10.1145/2046707.2046771Google ScholarDigital Library
- Ülkü Meteriz, Necip Fazil Yldran, Joongheon Kim, and David Mohaisen. 2020. Understanding the Potential Risks of Sharing Elevation Information on Fitness Applications. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS). 464--473. https://doi.org/10.1109/ICDCS47774.2020.00063Google Scholar
- J. V. Monaco. 2018. SoK: Keylogging Side Channels. In 2018 IEEE Symposium on Security and Privacy (SP). 211--228.Google Scholar
- Peter Norvig. 2007. https://norvig.com/spell-correct.htmlGoogle Scholar
- Nicole Odom, Jesse Lindmar, John Hirt, and Josh Brunty. 2019. Forensic Inspection of Sensitive User Data and Artifacts from Smartwatch Wearable Devices. Journal of Forensic Sciences, Vol. 64 (06 2019). https://doi.org/10.1111/1556--4029.14109Google ScholarCross Ref
- Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019).Google Scholar
- Mohd Sabra, Anindya Maiti, and Murtuza Jadliwala. 2020. Zoom on the Keystrokes: Exploiting Video Calls for Keystroke Inference Attacks. arxiv: 2010.12078 [cs.CR]Google Scholar
- Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, Vol. 15, 56 (2014), 1929--1958. http://jmlr.org/papers/v15/srivastava14a.htmlGoogle ScholarDigital Library
- T. Sztyler and H. Stuckenschmidt. 2017. Online personalization of cross-subjects based activity recognition models on wearable devices. In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom). 180--189. https://doi.org/10.1109/PERCOM.2017.7917864Google ScholarCross Ref
- Aku Visuri, Zhanna Sarsenbayeva, Niels van Berkel, Jorge Goncalves, Reza Rawassizadeh, Vassilis Kostakos, and Denzil Ferreira. 2017. Quantifying Sources and Types of Smartwatch Usage Sessions. Association for Computing Machinery, New York, NY, USA, 3569--3581. https://doi.org/10.1145/3025453.3025817Google ScholarDigital Library
- Tran Huy Vu, Archan Misra, Quentin Roy, Kenny Choo Tsu Wei, and Youngki Lee. 2018. Smartwatch-Based Early Gesture Detection 8 Trajectory Tracking for Interactive Gesture-Driven Applications. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, 1, Article 39 (March 2018), 27 pages. https://doi.org/10.1145/3191771Google ScholarDigital Library
- Martin Vuagnoux and Sylvain Pasini. 2009. Compromising Electromagnetic Emanations of Wired and Wireless Keyboards. USENIX Security Symposium (01 2009).Google Scholar
- Chen Wang, Xiaonan Guo, Yan Wang, Yingying Chen, and Bo Liu. 2016. Friend or Foe?: Your Wearable Devices Reveal Your Personal PIN. 189--200. https://doi.org/10.1145/2897845.2897847Google ScholarDigital Library
- He Wang, Ted Tsung-Te Lai, and Romit Roy Choudhury. 2015. MoLe: Motion Leaks through Smartwatch Sensors. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (Paris, France) (MobiCom '15). Association for Computing Machinery, New York, NY, USA, 155--166. https://doi.org/10.1145/2789168.2790121Google ScholarDigital Library
- X. Yu, Z. Zhou, M. Xu, X. You, and X. Li. 2020. ThumbUp: Identification and Authentication by Smartwatch using Simple Hand Gestures. In 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom). 1--10. https://doi.org/10.1109/PerCom45495.2020.9127367Google ScholarCross Ref
- Li Zhuang, Feng Zhou, and J. D. Tygar. 2009. Keyboard Acoustic Emanations Revisited. ACM Trans. Inf. Syst. Secur., Vol. 13, 1, Article 3 (Nov. 2009), 26 pages. https://doi.org/10.1145/1609956.1609959Google ScholarDigital Library
Index Terms
- SIA: Smartwatch-Enabled Inference Attacks on Physical Keyboards Using Acoustic Signals
Recommendations
Smartwatch-Based Keystroke Inference Attacks and Context-Aware Protection Mechanisms
ASIA CCS '16: Proceedings of the 11th ACM on Asia Conference on Computer and Communications SecurityWearable devices, such as smartwatches, are furnished with state-of-the-art sensors that enable a range of context-aware applications. However, malicious applications can misuse these sensors, if access is left unaudited. In this paper, we demonstrate ...
Poster Abstract: Mobile Context Logger: Recognizing User's Auditory Environments and Activities using Smartwatch
SenSys '23: Proceedings of the 21st ACM Conference on Embedded Networked Sensor SystemsSmartwatch, a wrist-worn personal device, can be a mobile platform for information assistant services that provide useful information in daily life, such as context-aware reminders for elderly people. To realize such an assistant service, we have ...
Function and Visual Experience Design Strategy of Chinese Elderly Health Monitoring Smartwatch
Design, User Experience, and UsabilityAbstractSince the 1990s, China’s ageing process has accelerated. With the increase of age and the ageing of physiological functions, the elderly are more prone to diseases. At present, there are not many health-monitoring smartwatches for the elderly in ...
Comments