skip to main content
10.1145/3338286.3340119acmconferencesArticle/Chapter ViewAbstractPublication PagesmobilehciConference Proceedingsconference-collections
research-article

Exploring Cross-Modal Training via Touch to Learn a Mid-Air Marking Menu Gesture Set

Published: 01 October 2019 Publication History

Abstract

While mid-air gestures are an attractive modality with an extensive research history, one challenge with their usage is that the gestures are not self-revealing. Scaffolding techniques to teach these gestures are difficult to implement since the input device, e.g. a hand, wand or arm, cannot present the gestures to the user. In contrast, for touch gestures, feedforward mechanisms (such as Marking Menus or OctoPocus) have been shown to effectively support user awareness and learning. In this paper, we explore whether touch gesture input can be leveraged to teach users to perform mid-air gestures. We show that marking menu touch gestures transfer directly to knowledge of mid-air gestures, allowing performance of these gestures without intervention. We argue that cross-modal learning can be an effective mechanism for introducing users to mid-air gestural input.

References

[1]
Christopher Ackad, Andrew Clayphan, Martin Tomitsch, and Judy Kay. 2015. An In-the-wild Study of Learning Mid-air Gestures to Browse Hierarchical Information at a Large Interactive Public Display. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 1227--1238. https://doi.org/10.1145/2750858.2807532
[2]
David Ahlström, Khalad Hasan, and Pourang Irani. 2014. Are You Comfortable Doing That?: Acceptance Studies of Around-device Gestures in and for Public Settings. In Proceedings of the 16th International Conference on Human-computer Interaction with Mobile Devices & Services (MobileHCI '14). ACM, New York, NY, USA, 193--202. https://doi.org/10.1145/2628363.2628381
[3]
Florian Alt, Sabrina Geiger, and Wolfgang Höhl. 2018. ShapelineGuide: Teaching Mid-Air Gestures for Large Interactive Displays. In Proceedings of the 7th ACM International Symposium on Pervasive Displays (PerDis '18). ACM, New York, NY, USA, Article 3, 8 pages. https://doi.org/10.1145/3205873.3205887
[4]
Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: Enhancing Movement Training with an Augmented Reality Mirror. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 311--320. https://doi.org/10.1145/2501988.2502045
[5]
Olivier Bau and Wendy E. Mackay. 2008. OctoPocus: A Dynamic Guide for Learning Gesture-based Command Sets. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 37--46. https://doi.org/10.1145/1449715.1449724
[6]
Thomas Baudel and Michel Beaudouin-Lafon. 1993. Charade: Remote Control of Objects Using Free-hand Gestures. Commun. ACM 36, 7 (July 1993), 28--35. https://doi.org/10.1145/159544.159562
[7]
Richard A Bolt. 1980. "Put-that-there": Voice and gesture at the graphics interface. Vol. 14. ACM.
[8]
Andrew Bragdon, Robert Zeleznik, Brian Williamson, Timothy Miller, and Joseph J. LaViola, Jr. 2009. GestureBar: Improving the Approachability of Gesture-based Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 2269--2278. https://doi.org/10.1145/1518701.1519050
[9]
Xiang Cao and Ravin Balakrishnan. 2004. VisionWand: Interaction Techniques for Large Displays Using a Passive Wand Tracked in 3D. ACM Trans. Graph. 23, 3 (Aug. 2004), 729--729. https://doi.org/10.1145/1015706.1015788
[10]
Christopher Clarke and Hans Gellersen. 2017. MatchPoint: Spontaneous Spatial Coupling of Body Movement for Touchless Pointing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 179--192. https://doi.org/10.1145/3126594.3126626
[11]
Andy Cockburn, Carl Gutwin, Joey Scarr, and Sylvain Malacria. 2014. Supporting Novice to Expert Transitions in User Interfaces. ACM Comput. Surv. 47, 2, Article 31 (Nov. 2014), 36 pages. https://doi.org/10.1145/2659796
[12]
William Delamare, Céline Coutrix, and Laurence Nigay. 2015. Designing Guiding Systems for Gesture-based Interaction. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '15). ACM, New York, NY, USA, 44--53. https://doi.org/10.1145/2774225.2774847
[13]
William Delamare, Thomas Janssoone, Céline Coutrix, and Laurence Nigay. 2016. Designing 3D Gesture Guidance: Visual Feedback and Feedforward Design Options. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '16). ACM, New York, NY, USA, 152--159. https://doi.org/10.1145/2909132.2909260
[14]
Tilman Dingler, Rufat Rzayev, Alireza Sahami Shirazi, and Niels Henze. 2018. Designing Consistent Gestures Across Device Types: Eliciting RSVP Controls for Phone, Watch, and Glasses. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 419, 12 pages. https://doi.org/10.1145/3173574.3173993
[15]
Dustin Freeman, Hrvoje Benko, Meredith Ringel Morris, and Daniel Wigdor. 2009. ShadowGuides: Visualizations for In-situ Learning of Multi-touch and Whole-hand Gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). ACM, New York, NY, USA, 165--172. https://doi.org/10.1145/1731903.1731935
[16]
Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2016. Do That, There: An Interaction Technique for Addressing In-Air Gesture Systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2319--2331. https://doi.org/10.1145/2858036.2858308
[17]
Sandra G. Hart and L E. Stavenland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. (12 1988), 139- pages. https://doi.org/10.1016/S0166-4115(08)62386-9
[18]
Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-air Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1063--1072. https://doi.org/10.1145/2556288.2557130
[19]
Mikkel R. Jakobsen, Yvonne Jansen, Sebastian Boring, and Kasper Hornbæk. 2015. Should I Stay or Should I Go? Selecting Between Touch and Mid-Air Gestures for Large-Display Interaction. In Human-Computer Interaction -- INTERACT 2015, Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing, Cham, 455--473.
[20]
Ankit Kamal, Yang Li, and Edward Lank. 2014. Teaching Motion Gestures via Recognizer Feedback. In Proceedings of the 19th International Conference on Intelligent User Interfaces (IUI '14). ACM, New York, NY, USA, 73--82. https://doi.org/10.1145/2557500.2557521
[21]
Keiko Katsuragawa, Ankit Kamal, Qi Feng Liu, Matei Negulescu, and Edward Lank. 2019. Bi-Level Thresholding: Analyzing the Effect of Repeated Errors in Gesture Input. ACM Trans. Interact. Intell. Syst. 9, 2-3, Article 15 (April 2019), 30 pages. https://doi.org/10.1145/3181672
[22]
Keiko Katsuragawa, Krzysztof Pietroszek, James R. Wallace, and Edward Lank. 2016. Watchpoint: Freehand Pointing with a Smartwatch in a Ubiquitous Display Environment. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '16). ACM, New York, NY, USA, 128--135. https://doi.org/10.1145/2909132.2909263
[23]
Gordon Kurtenbach and William Buxton. 1994. User Learning and Performance with Marking Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). ACM, New York, NY, USA, 258--264. https://doi.org/10.1145/191666.191759
[24]
Gordon Paul Kurtenbach. 1993. The Design and Evaluation of Marking Menus. Ph.D. Dissertation. Toronto, Ont., Canada, Canada. UMI Order No. GAXNN-82896.
[25]
Sarah Morrison-Smith, Megan Hofmann, Yang Li, and Jaime Ruiz. 2016. Using Audio Cues to Support Motion Gesture Interaction on Mobile Devices. ACM Trans. Appl. Percept. 13, 3, Article 16 (May 2016), 19 pages. https://doi.org/10.1145/2897516
[26]
Ian Oakley and Junseok Park. 2007. A Motion-based Marking Menu System. In CHI '07 Extended Abstracts on Human Factors in Computing Systems (CHI EA '07). ACM, New York, NY, USA, 2597--2602. https://doi.org/10.1145/1240866.1241048
[27]
Ian Oakley and Junseok Park. 2009. Motion Marking Menus: An Eyes-free Approach to Motion Input for Handheld Devices. Int. J. Hum.-Comput. Stud. 67, 6 (June 2009), 515--532. https://doi.org/10.1016/j.ijhcs.2009.02.002
[28]
Alexander Pacha. 2013. Sensor fusion for robust outdoor Augmented Reality tracking on mobile devices. (2013).
[29]
Krzyszt of Pietroszek, Anastasia Kuzminykh, James R. Wallace, and Edward Lank. 2014. Smartcasting: A Discount 3D Interaction Technique for Public Displays. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design (OzCHI '14). ACM, New York, NY, USA, 119--128. https://doi.org/10.1145/2686612.2686629
[30]
Otniel Portillo-Rodriguez, Oscar O. Sandoval-Gonzalez, Emanuele Ruffaldi, Rosario Leonardi, Carlo Alberto Avizzano, and Massimo Bergamasco. 2008. Real-Time Gesture Recognition, Evaluation and FeedForward Correction of a Multimodal Tai-Chi Platform. In Haptic and Audio Interaction Design, Antti Pirhonen and Stephen Brewster (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 30--39.
[31]
Gustavo Rovelo, Donald Degraen, Davy Vanacken, Kris Luyten, and Karin Coninx. 2015. Gestu-Wan - An Intelligible Mid-Air Gesture Guidance System for Walk-up-and-Use Displays. In INTERACT.
[32]
Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-defined Motion Gestures for Mobile Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 197--206. https://doi.org/10.1145/1978942.1978971
[33]
Joey Scarr, Andy Cockburn, Carl Gutwin, and Philip Quinn. 2011. Dips and Ceilings: Understanding and Supporting Transitions to Expertise in User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2741--2750. https://doi.org/10.1145/1978942.1979348
[34]
Christian Schönauer, Kenichiro Fukushi, Alex Olwal, Hannes Kaufmann, and Ramesh Raskar. 2012. Multimodal Motion Guidance: Techniques for Adaptive and Dynamic Feedback. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI '12). ACM, New York, NY, USA, 133--140. https://doi.org/10.1145/2388676.2388706
[35]
Shaishav Siddhpuria, Keiko Katsuragawa, James R. Wallace, and Edward Lank. 2017. Exploring At-Your-Side Gestural Interaction for Ubiquitous Environments. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS '17). ACM, New York, NY, USA, 1111--1122. https://doi.org/10.1145/3064663.3064695
[36]
Rajinder Sodhi, Hrvoje Benko, and Andrew Wilson. 2012. LightGuide: Projected Visualizations for Hand Movement Guidance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 179--188. https://doi.org/10.1145/2207676.2207702
[37]
Radu-Daniel Vatavu. 2012. Nomadic Gestures: A Technique for Reusing Gesture Commands for Frequent Ambient Interactions. J. Ambient Intell. Smart Environ. 4, 2 (April 2012), 79--93. http://dl.acm.org/citation.cfm?id=2350758.2350765
[38]
Radu-Daniel Vatavu. 2012. User-defined Gestures for Free-hand TV Control. In Proceedings of the 10th European Conference on Interactive TV and Video (EuroITV '12). ACM, New York, NY, USA, 45--48. https: //doi.org/10.1145/2325616.2325626
[39]
Radu-Daniel Vatavu. 2013. A Comparative Study of User-defined Handheld vs. Freehand Gestures for Home Entertainment Environments. J. Ambient Intell. Smart Environ. 5, 2 (March 2013), 187--211. http://dl.acm.org/citation.cfm?id=2594684.2594688
[40]
David Verweij, Vassilis-Javed Khan, Augusto Esteves, and Saskia Bakker. 2017. Multi-User Motion Matching Interaction for Interactive Television Using Smartwatches. In Adjunct Publication of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video (TVX '17 Adjunct). ACM, New York, NY, USA, 67--68. https://doi.org/10.1145/3084289.3089906
[41]
Robert Walter, Gilles Bailly, and Jörg Müller. 2013. StrikeAPose: Revealing Mid-air Gestures on Public Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 841--850. https://doi.org/10.1145/2470654.2470774
[42]
Andrew Wilson and Steven Shafer. 2003. XWand: UI for Intelligent Spaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, New York, NY, USA, 545--552. https://doi.org/10.1145/642611.642706
[43]
C. A. Wingrave, B. Williamson, P. D. Varcholik, J. Rose, A. Miller, E. Charbonneau, J. Bott, and J. J. LaViola. 2010. The Wiimote and Beyond: Spatially Convenient Devices for 3D User Interfaces. IEEE Computer Graphics and Applications 30, 2 (March 2010), 71--85. https://doi.org/10.1109/MCG.2009.109

Cited By

View all
  • (2024)Exploring Optimal Placement of Head-Based Hierarchical Marking Menus on SmartphonesMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63992-0_25(381-392)Online publication date: 19-Jul-2024
  • (2023)Pull Outperforms Push as Vibrotactile Wristband Feedback for Mid-Air Gesture GuidanceProceedings of Mensch und Computer 202310.1145/3603555.3603579(138-148)Online publication date: 3-Sep-2023
  • (2023)Ether-Mark: An Off-Screen Marking Menu For Mobile DevicesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614150(224-233)Online publication date: 9-Oct-2023
  • Show More Cited By

Index Terms

  1. Exploring Cross-Modal Training via Touch to Learn a Mid-Air Marking Menu Gesture Set

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MobileHCI '19: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services
    October 2019
    646 pages
    ISBN:9781450368254
    DOI:10.1145/3338286
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 01 October 2019

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Mobile interface
    2. Motion input
    3. Motor learning
    4. Training

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    MobileHCI '19
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 202 of 906 submissions, 22%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)13
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 24 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Exploring Optimal Placement of Head-Based Hierarchical Marking Menus on SmartphonesMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63992-0_25(381-392)Online publication date: 19-Jul-2024
    • (2023)Pull Outperforms Push as Vibrotactile Wristband Feedback for Mid-Air Gesture GuidanceProceedings of Mensch und Computer 202310.1145/3603555.3603579(138-148)Online publication date: 3-Sep-2023
    • (2023)Ether-Mark: An Off-Screen Marking Menu For Mobile DevicesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614150(224-233)Online publication date: 9-Oct-2023
    • (2022)Understanding and Creating Spatial Interactions with Distant Displays Enabled by Unmodified Off-The-Shelf SmartphonesMultimodal Technologies and Interaction10.3390/mti61000946:10(94)Online publication date: 19-Oct-2022
    • (2022)Investigating Cross-Modal Approaches for Evaluating Error Acceptability of a Recognition-Based Input TechniqueProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35172626:1(1-24)Online publication date: 29-Mar-2022
    • (2021)OctoPocus in VR: Using a Dynamic Guide for 3D Mid-Air Gestures in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.310185427:12(4425-4438)Online publication date: 1-Dec-2021
    • (2020)Depth and Breadth of Pie Menus for Mid-air Gesture InteractionInternational Journal of Human–Computer Interaction10.1080/10447318.2020.180924537:2(131-140)Online publication date: 26-Aug-2020

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media