Hao Jiang
ABSTRACT
This paper focuses on the real-life scenario that people are handwriting while wearing small mobile devices on their wrists. We explore the possibility of eavesdropping privacy-related information based on motion signals. To achieve this, we elaborately develop a new deep learning-based motion sensing framework with four major components, i.e., recorder, signal preprocessor, feature extractor and handwriting recognizer. First, we integrate a series of simple yet effective signal processing techniques to purify the sensory data to reflect the kinetic property of a handwriting motion. Then we take advantage of properties of Multimodal Convolutional Neural Network (MCNN) to extract abstract features. After that, a bidirectional Long Short-Term Memory (BLSTM) network is exploited to model temporal dynamics. Finally, we incorporate Connectionist Temporal Classification (CTC) algorithm to realize end-to-end handwriting recognition. We prototype our design using a commercial off-the-shelf smartwatch and carry out extensive experiments. The encouraging results reveal that our system can robustly achieve an average accuracy of 64% at character-level and 71.9% at word-level, and 56.6% accuracy rate for words unseen in the training set under certain conditions, which expose the danger of privacy disclosure in daily lives.
- Dmitri Asonov and Rakesh Agrawal. 2004. Keyboard Acoustic Emanations. In IEEE Symposium on Security & Privacy.Google Scholar
- Yigael Berger, Avishai Wool, and Arie Yeredor. 2006. Dictionary attacks using keyboard acoustic emanations. In Acm Conference on Computer & Communications Security.Google ScholarDigital Library
- Xu Chao, Parth H. Pathak, and Prasant Mohapatra. 2015. Finger-writing with Smartwatch:A Case for Finger and Hand Gesture Recognition using Smartwatch. (2015).Google Scholar
- Word Frequency Dat. 2016. Corpus of Contemporary American English. Retrieved March 13, 2016 from http://www.wordfrequency.infoGoogle Scholar
- Thomas Deselaers, Daniel Keysers, Jan Hosang, and Henry A. Rowley. 2015. GyroPen: Gyroscopes for Pen-Input With Mobile Phones. IEEE Transactions on Human-Machine Systems 45, 2 (2015), 263–271.Google ScholarCross Ref
- Haishi Du, Ping Li, Hao Zhou, Wei Gong, Gan Luo, and Panlong Yang. 2018. Wordrecorder: Accurate acoustic-based handwriting recognition using deep learning. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1448–1456.Google ScholarDigital Library
- Ahmed El-Sawy, Hazem El-Bakry, and Mohamed Loey. 2016. CNN for Handwritten Arabic Digits Recognition Based on LeNet-5. (2016).Google Scholar
- Alex Graves and Faustino Gomez. 2006. Connectionist temporal classification:labelling unsegmented sequence data with recurrent neural networks. In International Conference on Machine Learning.Google ScholarDigital Library
- Tzipora Halevi and Nitesh Saxena. 2014. Keyboard acoustic side channel attacks: exploring realistic and security-sensitive scenarios. International Journal of Information Security 14, 5 (2014), 1–14.Google Scholar
- Wang He, Tsung Te Lai, and Romit Roy Choudhury. 2015. MoLe: Motion Leaks through Smartwatch Sensors. In International Conference on Mobile Computing & Networking.Google Scholar
- S Hochreiter and J Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780.Google ScholarDigital Library
- Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In International Conference on International Conference on Machine Learning.Google Scholar
- Mobvoi AI Lab. 2012. Ticwatch. https://www.mobvoi.com/Google Scholar
- Chen Li, Wang Song, Fan Wei, Jun Sun, and Satoshi Naoi. 2015. Beyond human recognition: A CNN-based framework for handwritten character recognition. In Iapr Asian Conference on Pattern Recognition.Google Scholar
- Xinye Lin, Yixin Chen, Xiao Wen Chang, Xue Liu, and Xiaodong Wang. 2018. SHOW: Smart Handwriting on Watches. Proceedings of the Acm on Interactive Mobile Wearable & Ubiquitous Technologies 1(2018).Google ScholarDigital Library
- Anindya Maiti, Murtuza Jadliwala, Jibo He, and Igor Bilogrevic. 2015. (Smart)watch your taps:side-channel keystroke inference attacks using smartwatches. In Acm International Symposium on Wearable Computers.Google Scholar
- Arik Poznanski and Lior Wolf. 2016. CNN-N-Gram for HandwritingWord Recognition. (2016).Google Scholar
- Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. 2673–2681 pages.Google Scholar
- Qingxin Xia, Feng Hong, Yuan Feng, and Zhongwen Guo. 2018. MotionHacker: Motion sensor based eavesdropping on handwriting via smartwatch. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 468–473.Google ScholarCross Ref
- Xuefeng Xiao, Lianwen Jin, Yafeng Yang, Weixin Yang, Jun Sun, and Tianhai Chang. 2017. Building fast and compact convolutional neural networks for offline handwritten Chinese character recognition. Pattern Recognition 72(2017), 72–81.Google ScholarDigital Library
- Tuo Yu, Haiming Jin, and Klara Nahrstedt. 2016. WritingHacker:audio based eavesdropping of handwriting via mobile devices. In Acm International Joint Conference on Pervasive & Ubiquitous Computing.Google Scholar
- Maotian Zhang, Panlong Yang, Tian Chang, Shi Lei, Shaojie Tang, and Xiao Fu. 2015. SoundWrite:Text Input on Surfaces through Mobile Acoustic Sensing. In International Workshop on Experiences with the Design & Implementation of Smart Objects.Google ScholarDigital Library
- Li Zhuang, Feng Zhou, and J. D. Tygar. 2005. Keyboard acoustic emanations revisited. In Acm Conference on Computer & Communications Security.Google ScholarDigital Library
Recommendations
Style-preserving English handwriting synthesis
This paper presents a novel and effective approach to synthesize English handwriting in the user's writing style. We select the most important features that depict the handwriting style, including character glyph, size, slant, and pressure, special ...
Automatic generation of large-scale handwriting fonts via style learning
SA '16: SIGGRAPH ASIA 2016 Technical BriefsGenerating personal handwriting fonts with large amounts of characters is a boring and time-consuming task. Take Chinese fonts as an example, the official standard GB18030-2000 for commercial font products contains 27533 simplified Chinese characters. ...
Handwriting Recognition in Indian Regional Scripts: A Survey of Offline Techniques
Offline handwriting recognition in Indian regional scripts is an interesting area of research as almost 460 million people in India use regional scripts. The nine major Indian regional scripts are Bangla (for Bengali and Assamese languages), Gujarati, ...
Comments