skip to main content
10.1145/3242587.3242651acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

Touch+Finger: Extending Touch-based User Interface Capabilities with "Idle" Finger Gestures in the Air

Published: 11 October 2018 Publication History

Abstract

In this paper, we present Touch+Finger, a new interaction technique that augments touch input with multi-finger gestures for rich and expressive interaction. The main idea is that while one finger is engaged in a touch event, a user can leverage the remaining fingers, the "idle" fingers, to perform a variety of hand poses or in-air gestures to extend touch-based user interface capabilities. To fully understand the use of these idle fingers, we constructed a design space based on conventional touch gestures (i.e., single- and multi-touch gestures) and inter- action period (i.e., before and during touch). Considering the design space, we investigated the possible movement of the idle fingers and developed a total of 20 Touch+Finger gestures. Using ring-like devices to track the motion of the idle fingers in the air, we evaluated the Touch+Finger gestures on both recognition accuracy and ease of use. They were classified with a recognition accuracy of over 99% and received positive and negative comments from 8 participants. We suggested 8 interaction techniques with Touch+Finger gestures that demonstrate extended touch-based user interface capabilities.

Supplementary Material

suppl.mov (ufp1443p.mp4)
Supplemental video
MP4 File (p335-lim.mp4)

References

[1]
Shaikh Shawon Arefin Shimon, Courtney Lutton, Zichun Xu, Sarah Morrison-Smith, Christina Boucher, and Jaime Ruiz. 2016. Exploring non-touchscreen gestures for smartwatches. In Proceedings of the 2016 chi conference on human factors in computing systems. ACM, 3822--3833.
[2]
Sebastian Boring, David Ledo, Xiang 'Anthony' Chen, Nicolai Marquardt, Anthony Tang, and Saul Greenberg. 2012. The fat thumb: using the thumb's contact size for single-handed mobile interaction. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services. ACM, 39--48.
[3]
Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: multi-touch interaction around small devices. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, 201--204.
[4]
Edwin Chan, Teddy Seyed, Wolfgang Stuerzlinger, Xing-Dong Yang, and Frank Maurer. 2016. User elicitation on single-hand microgestures. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3403--3414.
[5]
Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E Hudson. 2014. Air
[6]
touch: interweaving touch & in-air gestures. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 519--525.
[7]
Matias Gonzalo Duarte, Daniel Marc Gatan Shiplacoff, Dianne Parry Dominguez, Jeremy Godfrey Lyon, Paul Mercer, and Peter Skillman. 2009. Extended touch-sensitive control area for electronic device. (Nov. 12 2009). US Patent App. 12/115,992.
[8]
John Greer Elias, Wayne Carl Westerman, and Myra Mary Haggerty. 2007. Multi-touch gesture dictionary. (Aug. 2 2007). US Patent App. 11/619,571.
[9]
Dustin Freeman, Hrvoje Benko, Meredith Ringel Morris, and Daniel Wigdor. 2009. ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 165--172.
[10]
Sarthak Ghosh, Hyeong Cheol Kim, Yang Cao, Arne Wessels, Simon T Perrault, and Shengdong Zhao. 2016. Ringteraction: Coordinated Thumb-index Interaction Using a Ring. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2640--2647.
[11]
Mark Hancock, Thomas Ten Cate, Sheelagh Carpendale, and Tobias Isenberg. 2010. Supporting sandtray therapy on an interactive tabletop. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2133--2142.
[12]
Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011. TapSense: enhancing finger interaction on touch surfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 627--636.
[13]
Ken Hinckley, Seongkook Heo, Michel Pahud, Christian Holz, Hrvoje Benko, Abigail Sellen, Richard Banks, Kenton O'Hara, Gavin Smyth, and William Buxton. 2016. Pre-touch sensing for mobile interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2869--2881.
[14]
Ken Hinckley and Hyunyoung Song. 2011. Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 801--810.
[15]
Motiv Inc. 2017. Motiv Ring. (2017). http://mymotiv.com/.
[16]
Bret Jackson, David Schroeder, and Daniel F Keefe. 2012. Nailing down multi-touch: anchored above the surface interaction for 3D modeling and navigation. In Proceedings of Graphics Interface 2012. Canadian Information Processing Society, 181--184.
[17]
Mikkel R Jakobsen, Yvonne Jansen, Sebastian Boring, and Kasper Hornbæk. 2015. Should I stay or should I go? Selecting between touch and mid-air gestures for large-display interaction. In Human-Computer Interaction. Springer, 455--473.
[18]
Lei Jing, Yinghui Zhou, Zixue Cheng, and Tongjun Huang. 2012. Magic ring: A finger-worn device for multiple appliances control using static finger gestures. Sensors 12, 5 (2012), 5775--5790.
[19]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 167--176.
[20]
G Julian Lepinski, Tovi Grossman, and George Fitzmaurice. 2010. The design and evaluation of multitouch marking menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2233--2242.
[21]
Oura Health Ltd. 2018. OURA. (2018). https://ouraring.com/.
[22]
Shahzad Malik, Abhishek Ranjan, and Ravin Balakrishnan. 2005. Interacting with large displays from a distance with vision-tracked multi-finger gestural input. In Proceedings of the 18th annual ACM symposium on User interface software and technology. ACM, 43--52.
[23]
Nicolai Marquardt, Ricardo Jota, Saul Greenberg, and Joaquim A Jorge. 2011. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In IFIP Conference on Human-Computer Interaction. Springer, 461--476.
[24]
Nicolai Marquardt, Johannes Kiemer, David Ledo, Sebastian Boring, and Saul Greenberg. 2011. Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 21--30.
[25]
myAHRS+. 2016. IMU sensor. (2016). Retrieved July 14, 2018 from note = https://github.com/withrobot/myAHRS_plus/tree/master/tutorial,.
[26]
Michael Nielsen, Moritz Störring, Thomas B Moeslund, and Erik Granum. 2003. A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In International gesture workshop. Springer, 409--420.
[27]
Shaikh Shawon Arefin Shimon, Sarah Morrison-Smith, Noah John, Ghazal Fahimi, and Jaime Ruiz. 2015. Exploring user-defined back-of-device gestures for mobile devices. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 227--232.
[28]
Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air gestures around unmodified mobile devices. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 319--329.
[29]
Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. 2015. Investigating the dexterity of multi-finger input for mid-air text entry. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3643--3652.
[30]
Feng Wang, Xiang Cao, Xiangshi Ren, and Pourang Irani. 2009. Detecting and leveraging finger orientation for interaction with direct-touch surfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. ACM, 23--32.
[31]
Feng Wang and Xiangshi Ren. 2009. Empirical evaluation for finger input properties in multi-touch interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1063--1072.
[32]
Hongyi Wen, Julian Ramos Rojas, and Anind K Dey. 2016. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3847--3851.
[33]
Wayne Westerman, John G Elias, and Alan Hedge. 2001. Multi-touch: A new tactile 2-d gesture interface for human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 45. SAGE Publications Sage CA: Los Angeles, CA, 632--636.
[34]
Gerard Wilkinson, Ahmed Kharrufa, Jonathan Hook, Bradley Pursglove, Gavin Wood, Hendrik Haeuser, Nils Y Hammerla, Steve Hodges, and Patrick Olivier. 2016. Expressy: Using a wrist-worn inertial measurement unit to add expressiveness to touch-based interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2832--2844.
[35]
Jacob O Wobbrock, Brad A Myers, and Htet Htet Aung. 2008. The performance of hand postures in front-and back-of-device interaction for mobile computing. International Journal of Human-Computer Studies 66, 12 (2008), 857--875.
[36]
Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S Reynolds. 2014. SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 527--534.
[37]
Jingjie Zheng and Daniel Vogel. 2016. Finger-Aware Shortcuts. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4274--4285.

Cited By

View all
  • (2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
  • (2023)Single-tap Latency Reduction with Single- or Double- tap PredictionProceedings of the ACM on Human-Computer Interaction10.1145/36042717:MHCI(1-26)Online publication date: 13-Sep-2023
  • (2023)Acoustic+Pose: Adding Input Modality to Smartphones with Near-Surface Hand-Pose Recognition using Acoustic SurfaceProceedings of the 2023 International Symposium on Wearable Computers10.1145/3594738.3611362(44-49)Online publication date: 8-Oct-2023
  • Show More Cited By

Index Terms

  1. Touch+Finger: Extending Touch-based User Interface Capabilities with "Idle" Finger Gestures in the Air

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology
    October 2018
    1016 pages
    ISBN:9781450359481
    DOI:10.1145/3242587
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 October 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. gestural interaction
    2. interaction techniques
    3. joint interaction
    4. touch input
    5. wearable rings

    Qualifiers

    • Research-article

    Conference

    UIST '18

    Acceptance Rates

    UIST '18 Paper Acceptance Rate 80 of 375 submissions, 21%;
    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)88
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking OnlyProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642323(1-18)Online publication date: 11-May-2024
    • (2023)Single-tap Latency Reduction with Single- or Double- tap PredictionProceedings of the ACM on Human-Computer Interaction10.1145/36042717:MHCI(1-26)Online publication date: 13-Sep-2023
    • (2023)Acoustic+Pose: Adding Input Modality to Smartphones with Near-Surface Hand-Pose Recognition using Acoustic SurfaceProceedings of the 2023 International Symposium on Wearable Computers10.1145/3594738.3611362(44-49)Online publication date: 8-Oct-2023
    • (2023)EFRingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694786:4(1-31)Online publication date: 11-Jan-2023
    • (2023)AO-Finger: Hands-free Fine-grained Finger Gesture Recognition via Acoustic-Optic Sensor FusingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581264(1-14)Online publication date: 19-Apr-2023
    • (2023)iFAD Gestures: Understanding Users’ Gesture Input Performance with Index-Finger Augmentation DevicesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580928(1-17)Online publication date: 19-Apr-2023
    • (2023)Around-device finger input on commodity smartwatches with learning guidance through discoverabilityInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103105179:COnline publication date: 1-Nov-2023
    • (2022)Metaphoraction: Support Gesture-based Interaction Design with Metaphorical MeaningsACM Transactions on Computer-Human Interaction10.1145/351189229:5(1-33)Online publication date: 20-Oct-2022
    • (2021)GestuRING: A Web-based Tool for Designing Gesture Input with Rings, Ring-Like, and Ring-Ready DevicesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474780(710-723)Online publication date: 10-Oct-2021
    • (2021)Interaction Illustration Taxonomy: Classification of Styles and Techniques for Visually Representing Interaction ScenariosProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445586(1-22)Online publication date: 6-May-2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media