Ruide Zhang
Skip Abstract Section
Abstract
With the increasing popularity of augmented reality (AR) services, providing seamless human-computer interactions in the AR setting has received notable attention in the industry. Gesture control devices have recently emerged to be the next great gadgets for AR due to their unique ability to enable computer interaction with day-to-day gestures. While these AR devices are bringing revolutions to our interaction with the cyber world, it is also important to consider potential privacy leakages from these always-on wearable devices. Specifically, the coarse access control on current AR systems could lead to possible abuse of sensor data.
Although the always-on gesture sensors are frequently quoted as a privacy concern, there has not been any study on information leakage of these devices. In this article, we present our study on side-channel information leakage of the most popular gesture control device, Myo. Using signals recorded from the electromyography (EMG) sensor and accelerometers on Myo, we can recover sensitive information such as passwords typed on a keyboard and PIN sequence entered through a touchscreen. EMG signal records subtle electric currents of muscle contractions. We design novel algorithms based on dynamic cumulative sum and wavelet transform to determine the exact time of finger movements. Furthermore, we adopt the Hudgins feature set in a support vector machine to classify recorded signal segments into individual fingers or numbers. We also apply coordinate transformation techniques to recover fine-grained spatial information with low-fidelity outputs from the sensor in keystroke recovery.
We evaluated the information leakage using data collected from a group of volunteers. Our results show that there is severe privacy leakage from these commodity wearable sensors. Our system recovers complex passwords constructed with lowercase letters, uppercase letters, numbers, and symbols with a mean success rate of 91%.
- Apple. 2017. Apple Watch Series 2—Technical Specifications. (2017). https://support.apple.com/kb/SP746?locale=en_US.Google Scholar
- Yousef Al-Assaf. 2006. Surface myoelectric signal analysis: Dynamic approaches for change detection and classification. IEEE Transactions on Biomedical Engineering 53, 11 (2006), 2248--2256. Google ScholarCross Ref
- Kamran Ali, Alex X. Liu, Wei Wang, and Muhammad Shahzad. 2015. Keystroke recognition using WiFi signals. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 90--102. Google ScholarDigital Library
- Dmitri Asonov and Rakesh Agrawal. 2004. Keyboard acoustic emanations. In Proceedings of the IEEE Symposium on Security and Privacy, Vol. 2004. 3--11. Google ScholarCross Ref
- Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, and Blair MacIntyre. 2001. Recent advances in augmented reality. IEEE Computer Graphics and Applications 21, 6 (2001), 34--47. Google ScholarDigital Library
- Myo Blog. 2015. Jake Sims is a straight-up wizard. Retrieved from http://developerblog.myo.com/featured-dev-jake-sims/.Google Scholar
- Bloomburg. 2016. Goldman Sachs has four charts showing the huge potential in virtual and augmented reality. Retrieved from http://www.bloomberg.com/news/articles/2016-01-13/goldman-sachs-has-four-charts-showing-the-huge-potential-in-virtual-and-augmented-reality.Google Scholar
- Davide Balzarotti, Marco Cova, and Giovanni Vigna. 2008. Clearshot: Eavesdropping on keyboard input from video. In Proceedings of the 2008 IEEE Symposium on Security and Privacy (sp 2008). IEEE, 170--183.Google ScholarDigital Library
- Mourad Barkat. 2005. Signal Detection and Estimation. Artech House.Google Scholar
- John V. Basmajian and C. J. De Luca. 1985. Muscles alive. Muscles Alive: Their Functions Revealed by Electromyography 278 (1985), 126.Google Scholar
- Yigael Berger, Avishai Wool, and Arie Yeredor. 2006. Dictionary attacks using keyboard acoustic emanations. In Proceedings of the 13th ACM Conference on Computer and Communications Security. ACM, 245--254. Google ScholarDigital Library
- CNET. 2016. Microsoft’s HoloLens is super limited—and hella magical. Retrieved from https://www.cnet.com/products/microsoft-hololens-hands-on/.Google Scholar
- Liang Cai and Hao Chen. 2011. TouchLogger: Inferring keystrokes on touch screen from smartphone motion. HotSec 11 (2011), 9--9.Google ScholarDigital Library
- Thomas P. Caudell and David W. Mizell. 1992. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proceedings of the 25th Hawaii International Conference on System Sciences, 1992, Vol. 2. IEEE, 659--669. Google ScholarCross Ref
- Francis H. Y. Chan, Yong-Sheng Yang, F. K. Lam, Yuan-Ting Zhang, and Philip A. Parker. 2000. Fuzzy EMG classification for prosthesis control. IEEE Transactions on Rehabilitation Engineering 8, 3 (2000), 305--311. Google ScholarCross Ref
- Charles K. Chui. 2014. An Introduction to Wavelets. Vol. 1. Academic Press.Google Scholar
- Enrico Costanza, Andreas Kunz, and Morten Fjeld. 2009. Mixed reality: A survey. In Human Machine Interaction. Springer, 47--68. Google ScholarDigital Library
- Carlo J. De Luca, Alexander Adam, Robert Wotiz, L. Donald Gilmore, and S. Hamid Nawab. 2006. Decomposition of surface EMG signals. Journal of Neurophysiology 96, 3 (2006), 1646--1657. Google ScholarCross Ref
- Cynthia Dwork. 2006. Differential privacy. In Proceedings of the 33rd International Colloquium on Automata, Languages and Programming, Part II (ICALP’06). Google ScholarDigital Library
- Wassim El Falou, Mohamad Khalil, and Jacques Duchene. 2000. AR-based method for change detection using dynamic cumulative sum. In Proceedings of the 7th IEEE Internattional Conference on Electronics, Circuits and Systems (ICECS), Vol. 1. 157--160. Google ScholarCross Ref
- Kevin Englehart, Bernard Hudgins, Philip A. Parker, and Maryhelen Stevenson. 1999. Classification of the myoelectric signal using time-frequency based representations. Medical Engineering 8 Physics 21, 6 (1999), 431--438.Google Scholar
- Fitbit. 2017. Fitbit Specs. Retrieved from https://www.fitbit.com/surge#specs.Google Scholar
- Craig L. Fancourt and Jose C. Principe. 2000. On the use of neural networks in the generalized likelihood ratio test for detecting abrupt changes in signals. In IJCNN (2). 243--252. Google ScholarCross Ref
- Kristin A. Farry, Ian D. Walker, and Richard G. Baraniuk. 1996. Myoelectric teleoperation of a complex robotic hand. IEEE Transactions on Robotics and Automation 12, 5 (1996), 775--788. Google ScholarCross Ref
- Steven Feiner, Blair MacIntyre, Tobias Höllerer, and Anthony Webster. 1997. A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies 1, 4 (1997), 208--217. Google ScholarCross Ref
- Earlence Fernandes, Justin Paupore, Amir Rahmati, Daniel Simionato, Mauro Conti, and Atul Prakash. 2016. Flowfence: Practical data protection for emerging IoT application frameworks. In Proceedings of the USENIX Security Symposium.Google ScholarDigital Library
- Olivia A. Grigg, V. T. Farewell, and D. J. Spiegelhalter. 2003. Use of risk-adjusted CUSUM and RSPRTcharts for monitoring in medical contexts. Statistical Methods in Medical Research 12, 2 (2003), 147--170. Google ScholarCross Ref
- Bernard Hudgins, Philip Parker, and Robert N. Scott. 1993. A new strategy for multifunction myoelectric control. IEEE Transactions on Biomedical Engineering 40, 1 (1993), 82--94. Google ScholarCross Ref
- Chuck Jorgensen, Diana D. Lee, and Shane Agabont. 2003. Sub auditory speech recognition based on EMG signals. In Proceedings of the International Joint Conference on Neural Networks, 2003, Vol. 4. IEEE, 3128--3133. Google ScholarCross Ref
- S. Sathiya Keerthi, Shirish Krishnaj Shevade, Chiranjib Bhattacharyya, and Karuturi Radha Krishna Murthy. 2001. Improvements to Platt’s SMO algorithm for SVM classifier design. Neural Computation 13, 3 (2001), 637--649. Google ScholarDigital Library
- Mohamad Khalil and Jacques Duchêne. 1999. Dynamic cumulative sum approach for change detection. IEEE Transactions on Signal Processing 47, 4 (1999), 1205.Google Scholar
- Mohamad Khalil and Jacques Duchêne. 2000. Uterine EMG analysis: A dynamic approach for change detection and classification. IEEE Transactions on Biomedical Engineering 47, 6 (2000), 748--756.Google ScholarCross Ref
- Robert Lewand. 2000. Cryptological Mathematics. MAA.Google Scholar
- Mengyuan Li, Yan Meng, Junyi Liu, Haojin Zhu, Xiaohui Liang, Yao Liu, and Na Ruan. 2016. When CSI meets public WiFi: Inferring your mobile phone password via wifi signals. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 1068--1079. Google ScholarDigital Library
- Xiangyu Liu, Zhe Zhou, Wenrui Diao, Zhou Li, and Kehuan Zhang. 2015. When good becomes evil: Keystroke inference with smartwatch. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. ACM, 1273--1285. Google ScholarDigital Library
- Myo. 2016. Homepage. Retrieved from https://www.myo.com/.Google Scholar
- Federico Maggi, Simone Gasparini, and Giacomo Boracchi. 2011. A fast eavesdropping attack against touchscreens. In Proceedings of the 2011 7th International Conference on Information Assurance and Security (IAS). IEEE, 320--325. Google ScholarCross Ref
- Anindya Maiti, Oscar Armbruster, Murtuza Jadliwala, and Jibo He. 2016. Smartwatch-based keystroke inference attacks and context-aware protection mechanisms. In Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security. ACM, 795--806. Google ScholarDigital Library
- Steve Mann. 1997. Wearable computing: A first step toward personal imaging. Computer 30, 2 (1997), 25--32. Google ScholarDigital Library
- Elaine Nicpon Marieb and Katja Hoehn. 2007. Human Anatomy 8 Physiology. Pearson Education.Google Scholar
- Philip Marquardt, Arunabh Verma, Henry Carter, and Patrick Traynor. 2011. (sp) iPhone: Decoding vibrations from nearby keyboards using mobile phone accelerometers. In Proceedings of the 18th ACM Conference on Computer and Communications Security. ACM, 551--562. Google ScholarDigital Library
- Emiliano Miluzzo, Alexander Varshavsky, Suhrid Balakrishnan, and Romit Roy Choudhury. 2012. Tapprints: Your finger taps have fingerprints. In Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services. ACM, 323--336. Google ScholarDigital Library
- Oussama Mustapha, Dimitri Lefebvre, Ghaleb Hoblos, Houcine Chafouk, and Mohamad Khalil. 2008a. Fault Detection Algorithm Based on Filters Bank Derived from Wavelet Packets. INTECH Open Access Publisher. Google ScholarCross Ref
- Oussama Mustapha, Dimitri Lefebvre, Mohamad Khalil, Ghaleb Hoblos, and Houcine Chafouk. 2008b. Filters bank derived from the wavelet transform for real time change detection in signal. In Proceedings of the 3rd International Conference on Information and Communication Technologies: From Theory to Applications (ICTTA’08). IEEE, 1--6. Google ScholarCross Ref
- George Papagiannakis, Gurminder Singh, and Nadia Magnenat-Thalmann. 2008. A survey of mobile and wireless technologies for augmented reality systems. Computer Animation and Virtual Worlds 19, 1 (2008), 3--22. Google ScholarDigital Library
- Ratatype. 2016. Learn how to touch type. Retrieved from http://www.ratatype.com/learn.Google Scholar
- DC Rainmaker. 2017. Hands-on: Garmins New Fenix 5 Multisport GPS Series with mapping! Retruecved from https://www.dcrainmaker.com/2017/01/hands-on-garmins-new-fenix-5-multisport-gps-serieswith-mapping.html.Google Scholar
- Rahul Raguram, Andrew M. White, Dibyendusekhar Goswami, Fabian Monrose, and Jan-Michael Frahm. 2011. iSpy: Automatic reconstruction of typed input from compromising reflections. In Proceedings of the 18th ACM Conference on Computer and Communications Security. ACM, 527--536. Google ScholarDigital Library
- Samsung. 2016. [In-Depth Look] The Parts and Pieces that Make the Gear S3 Tick. Retrieved from https://news.samsung.com/global/in-depth-look-the-parts-and-pieces-that-make-the-gear-s3-tick.Google Scholar
- Diksha Shukla, Rajesh Kumar, Abdul Serwadda, and Vir V. Phoha. 2014. Beware, your hands reveal your secrets!. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. ACM, 904--917. Google ScholarDigital Library
- Ivan E. Sutherland. 1968. A head-mounted three dimensional display. In Proceedings of the December 9--11, 1968, Fall Joint Computer Conference, Part I. ACM, 757--764. Google ScholarDigital Library
- G. Tsenov, A. H. Zeghbib, F. Palis, N. Shoylev, and V. Mladenov. 2006. Neural networks for online classification of hand and finger movements using surface EMG signals. In Proceedings of the 2006 8th Seminar on Neural Network Applications in Electrical Engineering. IEEE, 167--171. Google ScholarCross Ref
- D. W. F. Van Krevelen and R. Poelman. 2010. A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality 9, 2 (2010), 1.Google ScholarCross Ref
- Martin Vuagnoux and Sylvain Pasini. 2009. Compromising electromagnetic emanations of wired and wireless keyboards. In Proceedings of the USENIX Security Symposium. 1--16.Google Scholar
- Chen Wang, Xiaonan Guo, Yan Wang, Yingying Chen, and Bo Liu. 2016. Friend or foe?: Your wearable devices reveal your personal PIN. In Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security. ACM, 189--200. Google ScholarDigital Library
- He Wang, Ted Tsung-Te Lai, and Romit Roy Choudhury. 2015. Mole: Motion leaks through smartwatch sensors. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 155--166. Google ScholarDigital Library
- Zhi Xu, Kun Bai, and Sencun Zhu. 2012. Taplogger: Inferring user inputs on smartphone touchscreens using on-board motion sensors. In Proceedings of the 5th ACM Conference on Security and Privacy in Wireless and Mobile Networks. ACM, 113--124. Google ScholarDigital Library
- Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst. 2008. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 193--202.Google Scholar
- Tong Zhu, Qiang Ma, Shanfeng Zhang, and Yunhao Liu. 2014. Context-free attacks using keyboard acoustic emanations. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. ACM, 453--464. Google ScholarDigital Library
- Li Zhuang, Feng Zhou, and J. Doug Tygar. 2009. Keyboard acoustic emanations revisited. ACM Transactions on Information and System Security (TISSEC) 13, 1 (2009), 3.Google ScholarDigital Library
Index Terms
- From Electromyogram to Password: Exploring the Privacy Impact of Wearables in Augmented Reality
Comments