skip to main content
10.1145/3279778.3279799acmconferencesArticle/Chapter ViewAbstractPublication PagesissConference Proceedingsconference-collections
research-article

FingerInput: Capturing Expressive Single-Hand Thumb-to-Finger Microgestures

Published: 19 November 2018 Publication History

Abstract

Single-hand thumb-to-finger microgestures have shown great promise for expressive, fast and direct interactions. However, pioneering gesture recognition systems each focused on a particular subset of gestures. We are still in lack of systems that can detect the set of possible gestures to a fuller extent. In this paper, we present a consolidated design space for thumb-to-finger microgestures. Based on this design space, we present a thumb-to-finger gesture recognition system using depth sensing and convolutional neural networks. It is the first system that accurately detects the touch points between fingers as well as the finger flexion. As a result, it can detect a broader set of gestures than the existing alternatives, while also providing high-resolution information about the contact points. The system shows an average accuracy of 91% for the real-time detection of 8 demanding thumb-to-finger gesture classes. We demonstrate the potential of this technology via a set of example applications.

Supplementary Material

MP4 File (p177-soliman.mp4)

References

[1]
2015. NailO: Fingernails as an Input Surface. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15 (2015), 3015--3018.
[2]
Edwin Chan, Teddy Seyed, Wolfgang Stuerzlinger, Xing-Dong Yang, and Frank Maurer. 2016. User elicitation on single-hand microgestures. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3403--3414.
[3]
Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015a. Cyclopsring: Enabling whole-hand and context-aware interactions through a fisheye ring. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 549--556.
[4]
Liwei Chan, Chi-Hao Hsieh, Yi-Ling Chen, Shuo Yang, Da-Yuan Huang, Rong-Hao Liang, and Bing-Yu Chen. 2015b. Cyclops: Wearable and single-piece full-body gesture input devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3001--3009.
[5]
Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13 (2013), 255--260.
[6]
Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1504--1514.
[7]
Andrea Colaço, Ahmed Kirmani, Hye Soo Yang, Nan-Wei Gong, Chris Schmandt, and Vivek K Goyal. 2013. Mime: compact, low power 3D gesture sensing for interaction with head mounted displays. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 227--236.
[8]
Artem Dementyev and Joseph A Paradiso. 2014. WristFlex: low-power gesture input with wrist-worn pressure sensors. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 161--166.
[9]
Markus Funk, Sven Mayer, Michael Nistor, and Albrecht Schmidt. 2016. Mobile In-Situ Pick-by-Vision: Order Picking Support Using a Projector Helmet. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '16). ACM, New York, NY, USA, Article 45, 4 pages.
[10]
Sean G. Gustafson, Bernhard Rabe, and Patrick M. Baudisch. 2013. Understanding Palm-based Imaginary Interfaces: The Role of Visual and Tactile Cues when Browsing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, 889--898.
[11]
Chris Harrison, Hrvoje Benko, and Andrew D Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 441--450.
[12]
Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 453--462.
[13]
Christopher-Eyk Hrabia, Katrin Wolf, and Mathias Wilhelm. 2013. Whole hand modeling using 8 wearable sensors: biomechanics for hand pose prediction. In Proceedings of the 4th Augmented Human International Conference. ACM, 21--28.
[14]
Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1526--1537.
[15]
Da-Yuan Huang, Ming-Chang Tsai, Ying-Chao Tung, Min-Lun Tsai, Yen-Ting Yeh, Liwei Chan, Yi-Ping Hung, and Mike Y Chen. 2014. TouchSense: expanding touchscreen input vocabulary using different areas of users' finger pads. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 189--192.
[16]
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 675--678.
[17]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 167--176.
[18]
Eyal Krupka, Kfir Karmon, Noam Bloom, Daniel Freedman, Ilya Gurvich, Aviv Hurvitz, Ido Leichter, Yoni Smolin, Yuval Tzairi, Alon Vinnikov, and Aharon Bar Hillel. 2017. Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines. In Proceedings of ACM CHI. 14. https://sites.google.com/ site/aharonbarhillel/CHI_2017.pdf?attredirects=0
[19]
Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 142.
[20]
Christian Loclair, Sean Gustafson, and Patrick Baudisch. 2010. PinchWatch: a wearable device for one-handed microinteractions. In Proc. MobileHCI, Vol. 10.
[21]
Pranav Mistry, Pattie Maes, and Liyan Chang. 2009. WUW-wear Ur world: a wearable gestural interface. In CHI'09 extended abstracts on Human factors in computing systems. ACM, 4111--4116.
[22]
Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2017. Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor. In Proceedings of International Conference on Computer Vision (ICCV). http: //handtracker.mpi-inf.mpg.de/projects/OccludedHands/
[23]
Masa Ogata, Yuta Sugiura, Yasutoshi Makino, Masahiko Inami, and Michita Imai. 2013. SenSkin: adapting skin as a soft interface. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 539--544.
[24]
Masa Ogata, Yuta Sugiura, Hirotaka Osawa, and Michita Imai. 2012. iRing: Intelligent Ring Using Infrared Reflection. Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12 (2012), 131--136.
[25]
Alan Poston. 2000. Human engineering design data digest. Washington, DC: Department of Defense Human Factors Engineering Technical Advisory Group (2000).
[26]
Manuel Prätorius, Dimitar Valkov, Ulrich Burgbacher, and Klaus Hinrichs. 2014. DigiTap: an eyes-free VR/AR symbolic input device. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology. ACM, 9--18.
[27]
Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. 2014. Realtime and robust hand tracking from depth. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1106--1113.
[28]
Grégory Rogez, Maryam Khademi, JS Supancic III, Jose Maria Martinez Montiel, and Deva Ramanan. 2014. 3D hand pose detection in egocentric RGB-D images. In Workshop at the European Conference on Computer Vision. Springer, 356--371.
[29]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. CoRR abs/1505.04597 (2015). http://arxiv.org/abs/1505.04597
[30]
Kyeongeun Seo and Hyeonjoong Cho. 2014. AirPincher: a handheld device for recognizing delicate mid-air hand gestures. Proceedings of the adjunct publication of the 27th annual ACM symposium on User interface software and technology - UIST'14 Adjunct (2014), 83--84.
[31]
Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, and others. 2015. Accurate, robust, and flexible real-time hand tracking. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3633--3642.
[32]
Ayan Sinha, Chiho Choi, and Karthik Ramani. 2016. Deephand: Robust hand pose estimation by completing a matrix imputed with deep features. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4150--4158.
[33]
Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On- and Above-Skin Input Sensing through a Wearable Depth Sensor. In Proceedings of ACM CHI. 12. http: //handtracker.mpi-inf.mpg.de/projects/WatchSense/
[34]
Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. 2015. Fast and robust hand tracking using detection-guided optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3213--3221.
[35]
Carsten Stoll, Nils Hasler, Juergen Gall, Hans-Peter Seidel, and Christian Theobalt. 2011. Fast articulated motion tracking using a sums of gaussians body model. In Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 951--958.
[36]
Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2009. Brainy hand: an ear-worn hand gesture interaction device. In CHI'09 Extended Abstracts on Human Factors in Computing Systems. ACM, 4255--4260.
[37]
Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, and others. 2016. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM Transactions on Graphics (TOG) 35, 4 (2016), 143.
[38]
Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, and Shahram Izadi. 2017. Articulated distance fields for ultra-fast tracking of hands interacting. ACM Transactions on Graphics (TOG) 36, 6 (2017), 244.
[39]
Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-Meshes for Real-Time Hand Modeling and Tracking. ACM Transaction on Graphics (Proc. SIGGRAPH Asia) (2016).
[40]
Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics (ToG) 33, 5 (2014), 169.
[41]
Hsin-Ruey Tsai, Min-Chieh Hsiu, Jui-Chun Hsiao, Lee-Ting Huang, Mike Chen, and Yi-Ping Hung. 2016a. TouchRing. Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct - MobileHCI '16 (2016), 891--898.
[42]
Hsin-Ruey Tsai, Cheng-Yuan Wu, Lee-Ting Huang, and Yi-Ping Hung. 2016b. ThumbRing: private interactions using one-handed thumb motion input on finger segments. Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (2016), 791--798.
[43]
Hsin-ruey Tsai, Te-Yen Wu, Da-Yuan Huang, Min-Chieh Hsiu, Jui-Chun Hsiao, Yi-Ping Hung, Mike Y. Chen, and Bing-yu Chen. 2017. SegTouch: Enhancing Touch Input While Providing Touch Gestures on Screens Using Thumb-To-Index-Finger Gestures. Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '17 (2017), 2164--2171.
[44]
Koji Tsukadaa and Michiaki Yasumurab. 2001. Ubi-finger: Gesture input device for mobile use. In Ubicomp 2001 Informal Companion Proceedings. 11.
[45]
Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. 2016. Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation. International Journal of Computer Vision 118, 2 (01 Jun 2016), 172--193.
[46]
Dimitar Valkov, Ulrich Burgbacher, Klaus Hinrichs, and Computer Graphics. 2014. DigiTap: An Eyes-Free VR/AR Symbolic Input Device. (2014), 9--18.
[47]
Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. 2017. Crossing Nets: Combining GANs and VAEs With a Shared Latent Space for Hand Pose Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[48]
Saiwen Wang, Jie Song, Jamie Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST '16 (2016), 851--860.
[49]
Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. Iskin: flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2991--3000.
[50]
Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proceedings of the 35rd Annual ACM Conference on Human Factors in Computing Systems. ACM.
[51]
Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, and Ravi Karkar. 2017. DigiTouch : Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays. 1, 3 (2017), 1--21.
[52]
Christian Winkler, Julian Seifert, David Dobbelstein, and Enrico Rukzio. 2014. Pervasive Information Through Constant Personal Projection: The Ambient Mobile Pervasive Display (AMP-D). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 4117--4126.
[53]
Katrin Wolf, Anja Naumann, Michael Rohs, and Jörg Müller. 2011. A taxonomy of microinteractions: Defining microgestures based on ergonomic and scenario-Dependent requirements. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6946 LNCS, PART 1 (2011), 559--575.
[54]
Sang Ho Yoon, Ke Huo, Vinh P. Nguyen, and Karthik Ramani. 2015. TIMMi: Finger-worn Textile Input Device with Multimodal Sensing in Mobile Interaction. Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '14 (2015), 269--272.
[55]
Sang Ho Yoon, Ke Huo, and Karthik Ramani. 2014. Plex. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing Adjunct Publication - UbiComp '14 Adjunct (2014), 191--194.
[56]
Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012).
[57]
Cheng Zhang, Anandghan Waghmare, Pranav Kundra, Yiming Pu, Scott Gilliland, Thomas Ploetz, Thad E Starner, Omer T Inan, and Gregory D Abowd. 2017. FingerSound: Recognizing unistroke thumb gestures using a ring. 1, 3 (2017), 1--19.

Cited By

View all
  • (2024)Thumb-to-Finger Gesture Recognition Using COTS Smartwatch AccelerometersProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701600(184-195)Online publication date: 1-Dec-2024
  • (2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
  • (2024)Designing More Private and Socially Acceptable Hand-to-Face Gestures for Heads-Up ComputingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678994(637-639)Online publication date: 5-Oct-2024
  • Show More Cited By

Index Terms

  1. FingerInput: Capturing Expressive Single-Hand Thumb-to-Finger Microgestures

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ISS '18: Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces
      November 2018
      499 pages
      ISBN:9781450356947
      DOI:10.1145/3279778
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 November 2018

      Permissions

      Request permissions for this article.

      Check for updates

      Badges

      • Best Paper

      Author Tags

      1. depth sensor
      2. finger touch input
      3. gestural input
      4. gesture recognition
      5. microgesture
      6. on-body input
      7. skin input

      Qualifiers

      • Research-article

      Funding Sources

      • Cluster of Excellence
      • ERC

      Conference

      ISS '18
      Sponsor:

      Acceptance Rates

      ISS '18 Paper Acceptance Rate 28 of 105 submissions, 27%;
      Overall Acceptance Rate 147 of 533 submissions, 28%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)216
      • Downloads (Last 6 weeks)15
      Reflects downloads up to 11 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Thumb-to-Finger Gesture Recognition Using COTS Smartwatch AccelerometersProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701600(184-195)Online publication date: 1-Dec-2024
      • (2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
      • (2024)Designing More Private and Socially Acceptable Hand-to-Face Gestures for Heads-Up ComputingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678994(637-639)Online publication date: 5-Oct-2024
      • (2024)TouchpadAnyWear: Textile-Integrated Tactile Sensors for Multimodal High Spatial-Resolution Touch Inputs with Motion Artifacts ToleranceProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676344(1-14)Online publication date: 13-Oct-2024
      • (2024)Understanding Gesture and Microgesture Inputs for Augmented Reality MapsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661630(409-423)Online publication date: 1-Jul-2024
      • (2024)GraV: Grasp Volume Data for the Design of One-Handed XR InterfacesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661567(151-167)Online publication date: 1-Jul-2024
      • (2024)Designing Upper-Body Gesture Interaction with and for People with Spinal Muscular Atrophy in VRProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642884(1-19)Online publication date: 11-May-2024
      • (2024)STMG: A Machine Learning Microgesture Recognition System for Supporting Thumb-Based VR/AR InputProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642702(1-15)Online publication date: 11-May-2024
      • (2024)UI Mobility Control in XR: Switching UI Positionings between Static, Dynamic, and Self EntitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642220(1-12)Online publication date: 11-May-2024
      • (2024)Microgesture + Grasp: A journey from human capabilities to interaction with microgesturesInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103398(103398)Online publication date: Nov-2024
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media