Abstract
Smart wearable devices have become pervasive and are playing a more important role in our everyday lives. However, the small screen size and very few buttons make the interaction and control cumbersome and inconvenient. Previous solutions to mitigate this problem either require extra dedicated hardware, or instrument the user's fingers with special purpose sensors, limiting their real-life applications. We present iDial, a novel real time approach that enables a virtual dial plate on the hand back, extending the interaction beyond the small screen of wearable devices. iDial only employs the already built-in microphone and motion sensors of the commercial-off-the-shelf (COTS) device to facilitate interactions between user and wearable, without any extra hardware involved. The key idea is to exploit the acoustic signatures extracted from passive subtle acoustic signals to accurately recognize the virtual keys input on the skin of the hand back. We innovatively locate the virtual keys on the 4 pieces of metacarpal bones to significantly reduce the possibility of casual inputs. iDial also takes advantages of the motion sensor fusion already available inside the wearable to achieve robustness against the ambient noise and human voices efficiently. We design and implement iDial on the Samsung Gear S2 smartwatch. Our extensive experiments show that iDial is able to achieve an average recognition accuracy of 96.7%, and maintain high accuracies across varying user behaviors and different environments. iDial achieves a below 0.5s end-to-end latency with all the computations and processes happening at the cheap commodity wearable.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, iDial: Enabling a Virtual Dial Plate on the Hand Back for Around-Device Interaction
- 2016. Google Project Soli. (2016). https://www.atap.google.com/soli/Google Scholar
- 2017. Tizen. (2017). https://www.tizen.org/Google Scholar
- Dmitri Asonov and Rakesh Agrawal. 2004. Keyboard acoustic emanations. In 2012 IEEE Symposium on Security and Privacy. IEEE, 3.Google ScholarCross Ref
- Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002. Finite-time analysis of the multiarmed bandit problem. Machine learning 47, 2--3 (2002), 235--256. Google ScholarDigital Library
- Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing. Springer, 1--4.Google ScholarDigital Library
- Donald A Berry and Bert Fristedt. 1985. Bandit problems: sequential allocation of experiments (Monographs on statistics and applied probability). Springer.Google Scholar
- Steven F Boll and Dennis C Pulsipher. 1980. Suppression of acoustic noise in speech using two microphone adaptive noise cancellation. Acoustics, Speech and Signal Processing, IEEE Transactions on 28, 6 (1980), 752--753.Google Scholar
- Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: private and subtle interaction using fingertips. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 255--260. Google ScholarDigital Library
- Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2, 3 (2011), 27. Google ScholarDigital Library
- Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D input using two magnetic sensors. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 237--244. Google ScholarDigital Library
- Ke-Yu Chen, Shwetak Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1504--1514. Google ScholarDigital Library
- Jeremy Gummeson, Bodhi Priyantha, and Jie Liu. 2014. An energy harvesting wearable ring platform for gesture input on surfaces. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services (MobiSys). ACM, 162--175. Google ScholarDigital Library
- John Paulin Hansen, Florian Biermann, Janus Askø Madsen, Morten Jonassen, Haakon Lund, Javier San Agustin, and Sebastian Sztuk. 2015. A gaze interactive textual smartwatch interface. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium, on Wearable Computers. ACM, 839--847. Google ScholarDigital Library
- Chris Harrison, Hrvoje Benko, and Andrew D Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST). ACM, 441--450. Google ScholarDigital Library
- Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 453--462. Google ScholarDigital Library
- Christian Holz, Tovi Grossman, George Fitzmaurice, and Anne Agur. 2012. Implanted user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 503--512. Google ScholarDigital Library
- Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1526--1537. Google ScholarDigital Library
- Seungwoo Je, Brendan Rooney, Liwei Chan, and Andrea Bianchi. 2017. tactoRing: A Skin-Drag Discrete Display. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3106--3114. Google ScholarDigital Library
- Wolf Kienzle and Ken Hinckley. 2014. Lightring: Always-available 2D input on any surface. In Proceedings of the 21th annual ACM symposium on User interface software and technology. ACM, 157--160. Google ScholarDigital Library
- Sven Kratz and Michael Rohs. 2009. Hoverflow: exploring around-device interaction with IR distance sensors. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 42. Google ScholarDigital Library
- Gierad Laput, Xiao Robert, and Chris Harrison. 2017. ViBand: High-fidelity bio-acoustic sensing using commodity smartwatch. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 321--333. Google ScholarDigital Library
- Gierad Laput, Robert Xiao, Xiang'Anthony' Chen, Scott E Hudson, and Chris Harrison. 2014. Skin buttons: cheap, small, low-powered and clickable fixed-icon laser projectors. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 389--394. Google ScholarDigital Library
- Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang, and Marco Gruteser. 2015. Snooping Keystrokes with mm-level Audio Ranging on a Single Phone. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom). ACM, 142--154. Google ScholarDigital Library
- David JC MacKay. 2003. Information theory, inference and learning algorithms. Cambridge university press.Google Scholar
- Denys JC Matthies, Simon T Perrault, Bodo Urban, and Shengdong Zhao. 2015. Botential: Localizing On-Body Gestures by Measuring Electrical Signatures on the Human Skin. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 207--216. Google ScholarDigital Library
- Adiyan Mujibiya, Xiang Cao, Desney S Tan, Dan Morris, Shwetak N Patel, and Jun Rekimoto. 2013. The sound of touch: on-body touch and gesture sensing based on transdermal ultrasound propagation. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces. ACM, 189--198. Google ScholarDigital Library
- K Murty and Bayya Yegnanarayana. 2006. Combining evidence from residual phase and MFCC features for speaker recognition. Signal Processing Letters, IEEE 13, 1 (2006), 52--55.Google ScholarCross Ref
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1515--1525. Google ScholarDigital Library
- Shahriar Nirjon, Jeremy Gummeson, Dan Gelb, and Kyu-Han Kim. 2015. TypingRing: A Wearable Ring Platform for Text Input. In Proceedings of the 9th international conference on Mobile systems, applications, and services (MobiSys). ACM. Google ScholarDigital Library
- Simon T Perrault, Eric Lecolinet, James Eagan, and Yves Guiard. 2013. Watchit: simple gestures and eyes-free interaction for wristwatches and bracelets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1451--1460. Google ScholarDigital Library
- Manuel Prätorius, Aaron Scherzinger, and Klaus Hinrichs. 2015. SkInteract: An On-body Interaction System Based on Skin-Texture Recognition. In Human-Computer Interaction--INTERACT 2015. Springer, 425--432.Google Scholar
- Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a Smartwatch and I can Track my Users Arm. In Proceedings of the 14th annual international conference on Mobile systems, applications, and services. Google ScholarDigital Library
- Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On-and Above-Skin Input Sensing through a Wearable Depth Sensor. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3891--3902. Google ScholarDigital Library
- Yu-Chih Tung and Kang G Shin. 2015. EchoTag: Accurate Infrastructure-Free Indoor Location Tagging with Smartphones. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom). ACM, 525--536. Google ScholarDigital Library
- Vladimir Naumovich Vapnik and Vlamimir Vapnik. 1998. Statistical learning theory. Vol. 1. Wiley New York.Google Scholar
- He Wang, Ted Tsung-Te Lai, and Romit Roy Choudhury. 2015. Mole: Motion leaks through smartwatch sensors. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 155--166. Google ScholarDigital Library
- Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: virtual touch screen in the air using RF signals. In Proceedings of the 2014 ACM conference on SIGCOMM. ACM, 235--246. Google ScholarDigital Library
- Junjue Wang, Kaichen Zhao, Xinyu Zhang, and Chunyi Peng. 2014. Ubiquitous keyboard for small mobile devices: harnessing multipath fading for fine-grained keystroke localization. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services (MobiSys). ACM, 14--27. Google ScholarDigital Library
- Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. Iskin: flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2991--3000. Google ScholarDigital Library
- Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3095--3105. Google ScholarDigital Library
- Robert Xiao, Gierad Laput, and Chris Harrison. 2014. Expanding the input expressivity of smartwatches with mechanical pan, twist, tilt and click. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 193--196. Google ScholarDigital Library
- Chao Xu, Parth H Pathak, and Prasant Mohapatra. 2015. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 9--14. Google ScholarDigital Library
- Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a Mobile Device into a Mouse in the Air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys). ACM, 15--29. Google ScholarDigital Library
- Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1491--1503. Google ScholarDigital Library
- Tong Zhu, Qiang Ma, Shanfeng Zhang, and Yunhao Liu. 2014. Context-free Attacks Using Keyboard Acoustic Emanations. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 453--464. Google ScholarDigital Library
- Li Zhuang, Feng Zhou, and J Doug Tygar. 2009. Keyboard acoustic emanations revisited. ACM Transactions on Information and System Security (TISSEC) 13, 1 (2009), 3. Google ScholarDigital Library
Index Terms
- iDial: Enabling a Virtual Dial Plate on the Hand Back for Around-Device Interaction
Recommendations
EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristband
CHI '24: Proceedings of the CHI Conference on Human Factors in Computing SystemsOur hands serve as a fundamental means of interaction with the world around us. Therefore, understanding hand poses and interaction contexts is critical for human-computer interaction (HCI). We present EchoWrist, a low-power wristband that continuously ...
SkinTrack: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsSkinTrack is a wearable system that enables continuous touch tracking on the skin. It consists of a ring, which emits a continuous high frequency AC signal, and a sensing wristband with multiple electrodes. Due to the phase delay inherent in a high-...
Exploring Boundless Scroll by Extending Motor Space
MobileHCI '15: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and ServicesThis paper explores the Boundless Scroll interface that extends the motor space for finger drag motions into the off-screen space using a tracking system around the screen. Seamlessly extended motor spaces enable continuous scroll actions across the on- ...
Comments