skip to main content
10.1145/3412382.3458261acmconferencesArticle/Chapter ViewAbstractPublication PagescpsweekConference Proceedingsconference-collections
research-article

MagicInput: Training-free Multi-lingual Finger Input System using Data Augmentation based on MNISTs

Published: 20 May 2021 Publication History

Abstract

Text input systems based on device-free finger tracking technologies have attracted considerable attention in the use scenarios of mobile and the Internet-of-Things (IoT) devices. Issues pertaining to 2D tracking have prompted interest in using 1D finger trajectories for the recognition of handwritten letters. Nonetheless, 1D tracking imposes two major challenges: (i) Trajectory information loss from 2D to 1D; and (ii) Inter-user diversity in writing traits. These challenges could possibly be overcome by collecting a large training dataset for every user; however, this would impose an unacceptable burden on users. This paper presents a text input system with multi-language support without training using acoustic-based 1D finger tracking technology. We developed a novel data augmentation scheme, in which the handwritten image dataset MNISTs are used to create artificial datasets (called TrackMNISTs). We compensate for the trajectory information loss of 1D by creating personal dataset (from TrackMNIST) to match the writing habits of individual users. The proposed data augmentation mechanism is also applicable to multilingual letter recognition. In experiments, MagicInput achieved outstanding classification accuracy on unseen users: 10 digits (98.3%), 26 uppercase/lowercase English letters (97.8%/95.3%), 49 Japanese characters (91.4%), and the 30 commonly used Chinese characters (93.8%).

References

[1]
Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3d tracking via body radio reflections. In 11th {USENIX} Symposium on Networked Systems Design and Implementation ( {NSDI} 14). 317--329.
[2]
Baidu. 2021. Baidu Hanyu. https://hanyu.baidu.com/zici/. (2021).
[3]
Mingyu Chen, Ghassan AlRegib, and Biing-Hwang Juang. 2015. Air-writing recognition---Part I: Modeling and recognition of characters, words, and connecting motions. IEEE Transactions on Human-Machine Systems 46, 3 (2015), 403--413.
[4]
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. 2018. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718 (2018).
[5]
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, 2921--2926.
[6]
Thomas Deselaers, Daniel Keysers, Jan Hosang, and Henry A Rowley. 2014. Gyropen: Gyroscopes for pen-input with mobile phones. IEEE Transactions on Human-Machine Systems 45, 2 (2014), 263--271.
[7]
Haishi Du, Ping Li, Hao Zhou, Wei Gong, Gan Luo, and Panlong Yang. 2018. Wordrecorder: Accurate acoustic-based handwriting recognition using deep learning. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1448--1456.
[8]
Zhangjie Fu, Jiashuang Xu, Zhuangdi Zhu, Alex X Liu, and Xingming Sun. 2018. Writing in the air with WiFi signals for virtual reality devices. IEEE Transactions on Mobile Computing 18, 2 (2018), 473--484.
[9]
Zhengxin Guo, Fu Xiao, Biyun Sheng, Huan Fei, and Shui Yu. 2020. WiReader: Adaptive Air Handwriting Recognition Based on Commercial Wi-Fi Signal. IEEE Internet of Things Journal (2020).
[10]
Teach Handwriting. 2021. Print Letters Animations and Worksheets. https://www.teachhandwriting.co.uk/print-letters-beginners.html. (2021).
[11]
Kiran Joshi, Dinesh Bharadia, Manikanta Kotaru, and Sachin Katti. 2015. WiDeo: Fine-grained Device-free Motion Tracing using {RF} Backscatter. In 12th {USENIX} Symposium on Networked Systems Design and Implementation ( {NSDI} 15). 189--204.
[12]
Gierad Laput, Robert Xiao, Xiang' Anthony' Chen, Scott E Hudson, and Chris Harrison. 2014. Skin buttons: cheap, small, low-powered and clickable fixed-icon laser projectors. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 389--394.
[13]
Wenzhe Li and Tracy Hammond. 2011. Recognizing text through sound alone. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence. 1481--1486.
[14]
Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--19.
[15]
Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. 2011. CASIA online and offline Chinese handwriting databases. In 2011 International Conference on Document Analysis and Recognition. IEEE, 37--41.
[16]
Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. 2013. Online and offline handwritten Chinese character recognition: benchmarking on new databases. Pattern Recognition 46, 1 (2013), 155--162.
[17]
Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. 69--81.
[18]
Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1515--1525.
[19]
Masa Ogata, Yuta Sugiura, Hirotaka Osawa, and Michita Imai. 2012. iRing: intelligent ring using infrared reflection. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 131--136.
[20]
Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In Proceedings of the 14th international conference on frontiers in handwriting recognition. IEEE, 285--290.
[21]
Arik Poznanski and Lior Wolf. 2016. Cnn-n-gram for handwriting word recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2305--2314.
[22]
Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. Widraw: Enabling hands-free drawing in the air on commodity wifi devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 77--89.
[23]
Wei Wang, Alex X Liu, and Ke Sun. 2016. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. 82--94.
[24]
Teng Wei and Xinyu Zhang. 2015. mtrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 117--129.
[25]
Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. Iskin: flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2991--3000.
[26]
Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. Skinmarks: Enabling interactions on body landmarks using conformal skin electronics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 3095--3105.
[27]
Elliott Wen, Winston Seah, Bryan Ng, Xuefeng Liu, and Jiannong Cao. 2016. UbiTouch: ubiquitous smartphone touchpads using built-in proximity and ambient light sensors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 286--297.
[28]
Kaishun Wu, Qiang Yang, Baojie Yuan, Yongpan Zou, Rukhsana Ruby, and Mo Li. 2020. EchoWrite: An acoustic-based finger input system without training. IEEE Transactions on Mobile Computing (2020).
[29]
Yi-Chao Wu, Fei Yin, and Cheng-Lin Liu. 2017. Improving handwritten Chinese text recognition using neural network language models and convolutional neural network shape models. Pattern Recognition 65 (2017), 251--264.
[30]
Chao Xu, Parth H Pathak, and Prasant Mohapatra. 2015. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. 9--14.
[31]
Tuo Yu, Haiming Jin, and Klara Nahrstedt. 2016. Writinghacker: Audio based eavesdropping of handwriting via mobile devices. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 463--473.
[32]
Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-grained acoustic-based device-free tracking. In Proceedings of the 15th annual international conference on mobile systems, applications, and services. 15--28.
[33]
Maotian Zhang, Panlong Yang, Chang Tian, Lei Shi, Shaojie Tang, and Fu Xiao. 2015. Soundwrite: Text input on surfaces through mobile acoustic sensing. In Proceedings of the 1st International Workshop on Experiences with the Design and Implementation of Smart Objects. 13--17.

Cited By

View all
  • (2024)Enable Touch-based Communication between Laptop and SmartwatchCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677574(4-8)Online publication date: 5-Oct-2024
  • (2024)Sensing to Hear through MemoryProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595988:2(1-31)Online publication date: 15-May-2024
  • (2023)Handwriting Recognition System Leveraging Vibration Signal on SmartphonesIEEE Transactions on Mobile Computing10.1109/TMC.2022.314817222:7(3940-3951)Online publication date: 1-Jul-2023
  • Show More Cited By

Index Terms

  1. MagicInput: Training-free Multi-lingual Finger Input System using Data Augmentation based on MNISTs

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      IPSN '21: Proceedings of the 20th International Conference on Information Processing in Sensor Networks (co-located with CPS-IoT Week 2021)
      May 2021
      423 pages
      ISBN:9781450380980
      DOI:10.1145/3412382
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 20 May 2021

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Text input system
      2. multi-language support
      3. training-free

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      Conference

      IPSN '21
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 143 of 593 submissions, 24%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)26
      • Downloads (Last 6 weeks)4
      Reflects downloads up to 17 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Enable Touch-based Communication between Laptop and SmartwatchCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677574(4-8)Online publication date: 5-Oct-2024
      • (2024)Sensing to Hear through MemoryProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595988:2(1-31)Online publication date: 15-May-2024
      • (2023)Handwriting Recognition System Leveraging Vibration Signal on SmartphonesIEEE Transactions on Mobile Computing10.1109/TMC.2022.314817222:7(3940-3951)Online publication date: 1-Jul-2023
      • (2023)MagneComm+: Near-Field Electromagnetic Induction Communication With MagnetometerIEEE Transactions on Mobile Computing10.1109/TMC.2021.313348122:5(2789-2801)Online publication date: 1-May-2023
      • (2022)DoCamProceedings of the 28th Annual International Conference on Mobile Computing And Networking10.1145/3495243.3560523(405-418)Online publication date: 14-Oct-2022
      • (2022)A Cross-Domain Federated Learning Framework for Wireless Human SensingIEEE Network10.1109/MNET.001.220023136:5(122-128)Online publication date: Sep-2022

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media