skip to main content
10.1145/3132272.3134126acmconferencesArticle/Chapter ViewAbstractPublication PagesissConference Proceedingsconference-collections
research-article

Hand Contact Shape Recognition for Posture-Based Tabletop Widgets and Interaction

Published: 17 October 2017 Publication History

Abstract

Tabletop interaction can be enriched by considering whole hands as input instead of only fingertips. We describe a generalised, reproducible computer vision algorithm to recognise hand contact shapes, with support for arm rejection, as well as dynamic properties like finger movement and hover. A controlled experiment shows the algorithm can detect seven different contact shapes with roughly 91% average accuracy. The effect of long sleeves and non-user specific templates is also explored. The algorithm is used to trigger, parameterise, and dynamically control menu and tool widgets, and the usability of a subset of these are qualitatively evaluated in a realistic application. Based on our findings, we formulate a number of design recommendations for hand shape-based interaction.

References

[1]
Bailly, G., Demeure, A., Lecolinet, E. and Nigay, L. MultiTouch menu (MTM). In Proc. IHM 2008, 165--168.
[2]
Bailly, G., Müller, J. and Lecolinet, E. Design and evaluation of finger-count interaction: Combining multitouch gestures and menus. Int. Jour. of Human-Computer Studies, 70, 10 (2012), 673--689.
[3]
Bier, E. A., Stone, M. C., Pier, K., Buxton, W. and DeRose, T. D. Toolglass and magic lenses: the see-through interface. In Proc. SIGGRAPH '93, 73--80.
[4]
Brandl, P., Leitner, J., Seifried, T., Haller, M., Doray, B. and To, P. Occlusion-aware menu design for digital tabletops. In Proc. CHI EA 2009, 3223--3228.
[5]
Chen, X. A., Schwarz, J., Harrison, C., Mankoff, J. and Hudson, S. E. Air+touch: interweaving touch & in-air gestures. In Proc. UIST 2014, 519--525.
[6]
Epps, J., Lichman, S. and Wu, M. A study of hand shape use in tabletop gesture interaction. In Proc. CHI EA 2006, 748--753.
[7]
Foucault, C., Micaux, M., Bonnet, D. and Beaudouin-Lafon, M. SPad: a bimanual interaction technique for productivity applications on multi-touch tablets. In Proc. CHI EA 2014, 1879--1884.
[8]
Frati, V. and Prattichizzo, D. Using Kinect for hand tracking and rendering in wearable haptics. In Proc. WHC 2011, 317--321.
[9]
Ghomi, E., Huot, S., Bau, O., Beaudouin-Lafon, M. and Mackay, W. E. Arpège: learning multitouch chord gestures vocabularies. In Proc. ITS 2013, 209--218.
[10]
Gu, J., Han, J. and Lee, G. HandCall: Calling a Tool by a Hand Gesture on the Tabletop. In Proc. DIS 2012.
[11]
Guiard, Y. Asymmetric division of labor in human skilled bimanual action: the kinematic chain as a model. Journal of Motor Behavior, 19, 4 (1987), 486--517.
[12]
Harrison, C., Xiao, R., Schwarz, J. and Hudson, S. E. TouchTools: leveraging familiarity and skill with physical tools to augment touch interaction. In Proc. CHI 2014, 2913--2916.
[13]
Heo, S. and Lee, G. Indirect shear force estimation for multi-point shear force operations. In CHI 2013, 281--284.
[14]
Hinckley, K., Heo, S., Pahud, M., Holz, C., Benko, H., Sellen, A., Banks, R., O'Hara, K., Smyth, G. and Buxton, W. Pre-Touch Sensing for Mobile Interaction. In Proc. CHI 2016, 2869--2881.
[15]
Knies, R. In-Place: Interacting with Large Displays. Reporting on research by Pahud, M., Hinckley, K., and Buxton, B. TechNet Inside Microsoft Research Blog Post. 04.10.2012. https://blogs.technet.microsoft.com/inside_microsoft_research/2012/10/04/in-place-interacting-with-large-displays/
[16]
Koura, S., Suo, S., Kimura, A., Shibata, F. and Tamura, H. Amazing forearm as an innovative interaction device and data storage on tabletop display. In Proc. ITS 2012, 383--386.
[17]
Kurtenbach, G. P. The design and evaluation of marking menus. PhD Thesis, University of Toronto, 1993.
[18]
Le, K.-D., Paknezhad, M., Woźniak, P. W., Azh, M., Kasparavičiūté, G., Fjeld, M., Zhao, S. and Brown, M. S. Towards Leaning Aware Interaction with Multitouch Tabletops. In Proc. NordiCHI '16, 1--4.
[19]
Luo, Y. and Vogel, D. Pin-and-Cross: A Unimanual Multitouch Technique Combining Static Touches with Crossing Selection. In Proc. UIST 2015, 323--332.
[20]
Matulic, F. and Norrie, M. Pen and Touch Gestural Environment for Document Editing on Interactive Tabletops. In Proc. ITS 2013, 41--50.
[21]
Ming-Kuei, H. Visual pattern recognition by moment invariants. IEEE Trans. Inf. Theory, 8, 2 (1962), 179--187.
[22]
Moghaddam, A. B., Svendsen, J., Tory, M. and Albu, A. B. Integrating touch and near touch interactions for information visualizations. In Proc. CHI EA 2011, 2347--2352.
[23]
Pyryeskin, D., Hancock, M. and Hoey, J. Extending interactions into hoverspace using reflected light. In Proc. ITS 2011, 262--263.
[24]
Schmidt, D., Chong, M. K. and Gellersen, H. HandsDown: hand-contour-based user identification for interactive surfaces. In Proc. NordiCHI 2010, 432--441.
[25]
Schwarz, J., Xiao, R., Mankoff, J., Hudson, S. E. and Harrison, C. Probabilistic palm rejection using spatiotemporal touch features and iterative classification. In Proc. CHI 2014, 2009--2012.
[26]
Uddin, M. S., Gutwin, C. and Lafreniere, B. HandMark Menus: Rapid Command Selection and Large Command Sets on Multi-Touch Displays. In Proc. CHI 2016, 5836--5848.
[27]
Vogel, D. and Casiez, G. Hand occlusion on a multi-touch tabletop. In Proc. CHI 2012, 2307--2316.
[28]
Westerman, W. Hand tracking, finger identification, and chordic manipulation on a multi-touch surface. PhD Thesis, University of Delaware, 1999.
[29]
Wigdor, D., Benko, H., Pella, J., Lombardo, J. and Williams, S. Rock & rails: extending multi-touch interactions with shape gestures to enable precise spatial manipulations. In Proc. CHI 2011, 1581--1590.
[30]
Wu, M. and Balakrishnan, R. Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In Proc. UIST 2003, 193--202.
[31]
Wu, M., Chia, S., Ryall, K., Forlines, C. and Balakrishnan, R. Gesture registration, relaxation, and reuse for multi-point direct-touch surfaces. In Proc. TABLETOP 2006, 185--192.
[32]
Xiang, C., Wilson, A. D., Balakrishnan, R., Hinckley, K. and Hudson, S. E. ShapeTouch: Leveraging contact shape on interactive surfaces. In Proc. TABLETOP 2008, 129--136.
[33]
Xiao, R., Schwarz, J. and Harrison, C. Estimating 3D Finger Angle on Commodity Touchscreens. In Proc. ITS 2015, 47--50.
[34]
Yoshikawa, T., Shizuki, B. and Tanaka, J. HandyWidgets: local widgets pulled-out from hands. In Proc. ITS 2012, 197--200.
[35]
Zhang, Z., Zhang, F., Chen, H., Liu, J., Wang, H. and Dai, G. Left and right hand distinction for multi-touch tabletop interactions. In Proc. IUI 2014, 47--56.

Cited By

View all
  • (2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
  • (2023)Spreadsheets on Interactive Surfaces: Breaking through the Grid with the PenACM Transactions on Computer-Human Interaction10.1145/363009731:2(1-33)Online publication date: 25-Oct-2023
  • (2023)Single-tap Latency Reduction with Single- or Double- tap PredictionProceedings of the ACM on Human-Computer Interaction10.1145/36042717:MHCI(1-26)Online publication date: 13-Sep-2023
  • Show More Cited By

Index Terms

  1. Hand Contact Shape Recognition for Posture-Based Tabletop Widgets and Interaction

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ISS '17: Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces
    October 2017
    504 pages
    ISBN:9781450346917
    DOI:10.1145/3132272
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 17 October 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. bimanual input
    2. interactive tabletops
    3. multi-touch
    4. palm rejection
    5. pen+touch
    6. shape recognition

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    ISS '17
    Sponsor:
    ISS '17: Interactive Surfaces and Spaces
    October 17 - 20, 2017
    Brighton, United Kingdom

    Acceptance Rates

    ISS '17 Paper Acceptance Rate 32 of 119 submissions, 27%;
    Overall Acceptance Rate 147 of 533 submissions, 28%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)14
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
    • (2023)Spreadsheets on Interactive Surfaces: Breaking through the Grid with the PenACM Transactions on Computer-Human Interaction10.1145/363009731:2(1-33)Online publication date: 25-Oct-2023
    • (2023)Single-tap Latency Reduction with Single- or Double- tap PredictionProceedings of the ACM on Human-Computer Interaction10.1145/36042717:MHCI(1-26)Online publication date: 13-Sep-2023
    • (2022)Keep in Touch: Combining Touch Interaction with Thumb-to-Finger µGestures for People with Visual ImpairmentProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556589(105-116)Online publication date: 7-Nov-2022
    • (2021)A Design Space for User Interface Elements using Finger Orientation InputProceedings of Mensch und Computer 202110.1145/3473856.3473862(1-10)Online publication date: 5-Sep-2021
    • (2021)TypeBoard: Identifying Unintentional Touch on Pressure-Sensitive Touchscreen KeyboardsThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474770(568-581)Online publication date: 10-Oct-2021
    • (2021)3D Hand Pose Estimation on Conventional Capacitive TouchscreensProceedings of the 23rd International Conference on Mobile Human-Computer Interaction10.1145/3447526.3472045(1-13)Online publication date: 27-Sep-2021
    • (2021)Attentive Sequence-to-Sequence Modeling of Stroke Gestures Articulation PerformanceIEEE Transactions on Human-Machine Systems10.1109/THMS.2021.311296151:6(663-672)Online publication date: Dec-2021
    • (2021)Deep Learning-Based Hand Posture Recognition for Pen Interaction EnhancementArtificial Intelligence for Human Computer Interaction: A Modern Approach10.1007/978-3-030-82681-9_7(193-225)Online publication date: 5-Nov-2021
    • (2020)Imprint-Based Input Techniques for Touch-Based Mobile DevicesProceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia10.1145/3428361.3428393(32-41)Online publication date: 22-Nov-2020
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media