Adina Friedman
Francesco Cafaro
ABSTRACT
Interactive public displays are now pervasive; however, designing gestures to interact with them is still a challenge for embodied interaction. We introduce a methodology, based on Conceptual Metaphor Theory (CMT) and Elicitation Studies, that can inform the work of interaction designers when they craft gestures that "make sense" to their users. Our approach is structured in three phases: elicitation, analysis, and design. Through the use case of an interactive video player, we describe how we collected data from participants in an elicitation study, how we extracted the embodied schemata that users associated with each video function, and how an interaction designer can use those schemata to design multiple gestures. We then present the results of a survey that show that the gestures generated with embodied schemata made more sense to people than those crafted without using embodied schemata.
Supplemental Material
- A’aeshah Alhakamy, Milka Trajkova, and Francesco Cafaro. 2021. Show Me How You Interact, I Will Tell You What You Think: Exploring the Effect of the Interaction Style on Users’ Sensemaking about Correlation and Causation in Data. Designing Interactive Systems Conference 2021, 564–575. https://doi.org/10.1145/3461778.3462083Google Scholar
Digital Library
- Abdullah Ali, Meredith Ringel Morris, and Jacob O. Wobbrock. 2021. “I Am Iron Man”: Priming Improves the Learnability and Memorability of User-Elicited Gestures. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 359, 14 pages. https://doi.org/10.1145/3411764.3445758Google ScholarDigital Library
- Abdullah X. Ali, Meredith Ringel Morris, and Jacob O. Wobbrock. 2019. Crowdlicit: A System for Conducting Distributed End-User Elicitation and Identification Studies. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300485Google ScholarDigital Library
- Florian Alt and Julia Vehns. 2016. Opportunistic Deployments: Challenges and Opportunities of Conducting Public Display Research at an Airport. In Proceedings of the 5th ACM International Symposium on Pervasive Displays (Oulu, Finland) (PerDis ’16). Association for Computing Machinery, New York, NY, USA, 106–117. https://doi.org/10.1145/2914920.2915020Google ScholarDigital Library
- Alissa N Antle, Greg Corness, and Milena Droumeva. 2009. Human-computer-intuition? Exploring the cognitive basis for intuition in embodied interaction. International Journal of Arts and Technology 2, 3 (2009), 235–254.Google ScholarCross Ref
- Alissa N Antle, Milena Droumeva, and Greg Corness. 2008. Playing with the sound maker: do embodied metaphors help children learn?. In Proceedings of the 7th international conference on Interaction design and children. 178–185.Google ScholarDigital Library
- Cordula Baur, Carolin Wienrich, and Jörn Hurtienne. 2022. Form follows mental models: Finding instantiations of image schemas using a design research approach. In Designing Interactive Systems Conference. 586–598.Google ScholarDigital Library
- Philip Bell, Bruce Lewenstein, Andrew W Shouse, Michael A Feder, 2009. Learning science in informal environments: People, places, and pursuits. Vol. 140. National Academies Press Washington, DC.Google Scholar
- Francesco Cafaro, Leilah Lyons, and Alissa N Antle. 2018. Framed guessability. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC Canada). ACM, New York, NY, USA.Google ScholarDigital Library
- Francesco Cafaro, Leilah Lyons, Jessica Roberts, and Josh Radinsky. 2014. The uncanny valley of embodied interaction design. Proceedings of the 2014 conference on Designing interactive systems. https://doi.org/10.1145/2598510.2598593Google ScholarDigital Library
- Francesco Cafaro, Alessandro Panella, Leilah Lyons, Jessica Roberts, and Josh Radinsky. 2013. I see you there!Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1911–1920. https://doi.org/10.1145/2470654.2466252Google ScholarDigital Library
- Francesco Cafaro and Jessica Roberts. 2021. Data through Movement: Designing Embodied Human-Data Interaction for Informal Learning. Synthesis Lectures on Visualization 8, 3 (2021), 1–127.Google ScholarCross Ref
- Antoine Clarinval, Anthony Simonofski, Benoît Vanderose, and Bruno Dumas. 2020. Public displays and citizen participation: a systematic literature review and research agenda. Transforming Government: People, Process and Policy (2020).Google Scholar
- Haiwei Dong, Ali Danesh, Nadia Figueroa, and Abdulmotaleb El Saddik. 2015. An elicitation study on gesture preferences and memorability toward a practical hand-gesture vocabulary for smart televisions. IEEE access 3 (2015), 543–555.Google ScholarCross Ref
- Paul Dourish. 2001. Where the action is. MIT press Cambridge.Google Scholar
- Niklas Elmqvist. 2011. Embodied human-data interaction. In ACM CHI 2011 Workshop “Embodied Interaction: Theory and Practice in HCI, Vol. 1. 104–107.Google Scholar
- Saul Greenberg, Sebastian Boring, Jo Vermeulen, and Jakub Dostal. 2014. Dark patterns in proxemic interactions. Proceedings of the 2014 conference on Designing interactive systems, 523–532. https://doi.org/10.1145/2598510.2598541Google ScholarDigital Library
- Saul Greenberg, Nicolai Marquardt, Till Ballendat, Rob Diaz-Marino, and Miaosen Wang. 2011. Proxemic interactions. Interactions 18 (1 2011), 42–50. Issue 1. https://doi.org/10.1145/1897239.1897250Google ScholarDigital Library
- Lynn Hoff, Eva Hornecker, and Sven Bertel. 2016. Modifying Gesture Elicitation: Do Kinaesthetic Priming and Increased Production Reduce Legacy Bias?. In Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (Eindhoven, Netherlands) (TEI ’16). Association for Computing Machinery, New York, NY, USA, 86–91. https://doi.org/10.1145/2839462.2839472Google ScholarDigital Library
- Eva Hornecker. 2011. The role of physicality in tangible and embodied interactions. interactions 18, 2 (2011), 19–23.Google Scholar
- Jörn Hurtienne. 2017. How cognitive linguistics inspires HCI: image schemas and image-schematic metaphors. International Journal of Human–Computer Interaction 33, 1 (2017), 1–20.Google ScholarCross Ref
- Jörn Hurtienne and Johann Habakuk Israel. 2007. Image schemas and their metaphorical extensions. Proceedings of the 1st international conference on Tangible and embedded interaction - TEI ’07, 127. https://doi.org/10.1145/1226969.1226996Google ScholarDigital Library
- Jörn Hurtienne, Johann Habakuk Israel, and Katharina Weber. 2008. Cooking up real world business applications combining physicality, digitality, and image schemas. In Proceedings of the 2nd international conference on Tangible and embedded interaction. 239–246.Google ScholarDigital Library
- J. Hurtienne, K. Klockner, S. Diefenbach, C. Nass, and A. Maier. 2015. Designing with Image Schemas: Resolving the Tension Between Innovation, Inclusion and Intuitive Use. Interacting with Computers 27 (5 2015), 235–255. Issue 3. https://doi.org/10.1093/iwc/iwu049Google ScholarCross Ref
- Jörn Hurtienne, Christian Stößel, Christine Sturm, Alexander Maus, Matthias Rötting, Patrick Langdon, and John Clarkson. 2010. Physical gestures for abstract concepts: Inclusive design with primary metaphors. Interacting with Computers 22, 6 (2010), 475–484.Google ScholarDigital Library
- Iraide Ibarretxe-Antuñano. 2013. The relationship between conceptual metaphor and culture. Intercultural pragmatics 10, 2 (2013), 315–339.Google Scholar
- Mark Johnson. 2013. The body in the mind: The bodily basis of meaning, imagination, and reason. University of Chicago press.Google Scholar
- Brigitte Jordan and Austin Henderson. 1995. Interaction analysis: Foundations and practice. The journal of the learning sciences 4, 1 (1995), 39–103.Google Scholar
- Bernhard Klein. 2015. A Gesture Control Framework Targeting High-Resolution Video Wall Displays. 2015 19th International Conference on Information Visualisation, 366–371. https://doi.org/10.1109/iV.2015.70Google ScholarDigital Library
- Panayiotis Koutsabasis and Panagiotis Vogiatzidakis. 2019. Empirical research in mid-air interaction: A systematic review. International Journal of Human–Computer Interaction 35, 18 (2019), 1747–1768.Google ScholarCross Ref
- George Lakoff. 2009. The neural theory of metaphor. Available at SSRN 1437794 (2009).Google Scholar
- George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press.Google Scholar
- Anna Macaranas, Alissa N Antle, and Bernhard E Riecke. 2012. Bridging the gap: Attribute and spatial metaphors for tangible interface design. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction. 161–168.Google ScholarDigital Library
- David McNeill. 1992. Hand and mind: What gestures reveal about thought. University of Chicago press.Google Scholar
- Swati Mishra and Francesco Cafaro. 2018. Full body interaction beyond fun: Engaging museum visitors in human-data interaction. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction. 313–319.Google ScholarDigital Library
- Meredith Ringel Morris. 2012. Web on the wall: insights from a multimodal interaction elicitation study. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces. 95–104.Google ScholarDigital Library
- Meredith Ringel Morris, Andreea Danielescu, Steven Drucker, Danyel Fisher, Bongshin Lee, MC Schraefel, and Jacob O Wobbrock. 2014. Reducing legacy bias in gesture elicitation studies. interactions 21, 3 (2014), 40–45.Google ScholarDigital Library
- Mathieu Nancel, Olivier Chapuis, Emmanuel Pietriga, Xing-Dong Yang, Pourang P. Irani, and Michel Beaudouin-Lafon. 2013. High-precision pointing on large wall displays using small handheld devices. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 831–840. https://doi.org/10.1145/2470654.2470773Google ScholarDigital Library
- Toni Pakkanen, Jaakko Hakulinen, Tero Jokela, Ismo Rakkolainen, Jari Kangas, Petri Piippo, Roope Raisamo, and Marja Salmimaa. 2017. Interaction with WebVR 360° video player: Comparing three interaction paradigms. 2017 IEEE Virtual Reality (VR), 279–280. https://doi.org/10.1109/VR.2017.7892285Google ScholarCross Ref
- Callum Parker, Martin Tomitsch, Nigel Davies, Nina Valkanova, and Judy Kay. 2020. Foundations for Designing Public Interactive Displays That Provide Value to Users. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376532Google ScholarDigital Library
- Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst, and Andy Cockburn. 2013. User-defined gestures for augmented reality. In IFIP Conference on Human-Computer Interaction. Springer, 282–299.Google ScholarDigital Library
- Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-defined motion gestures for mobile interaction. In Proceedings of the SIGCHI conference on human factors in computing systems. 197–206.Google ScholarDigital Library
- S Bruce Thomson. 2010. Sample size and grounded theory. Thomson, SB (2010). Grounded Theory-Sample Size. Journal of Administration and Governance 5, 1 (2010), 45–52.Google Scholar
- Milka Trajkova, A’aeshah Alhakamy, Francesco Cafaro, Rashmi Mallappa, and Sreekanth R. Kankara. 2020. Move Your Body: Engaging Museum Visitors with Human-Data Interaction. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376186Google ScholarDigital Library
- Rolf Ulrich and Claudia Maienborn. 2010. Left–right coding of past and future in language: The mental timeline during sentence processing. Cognition 117 (11 2010), 126–138. Issue 2. https://doi.org/10.1016/j.cognition.2010.08.001Google ScholarCross Ref
- Radu-Daniel Vatavu and Ionut-Alexandru Zaiti. 2014. Leap gestures for TV: insights from an elicitation study. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video. 131–138.Google ScholarDigital Library
- Santiago Villarreal-Narvaez, Jean Vanderdonckt, Radu-Daniel Vatavu, and Jacob O Wobbrock. 2020. A systematic review of gesture elicitation studies: What can we learn from 216 studies?. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 855–872.Google ScholarDigital Library
- Robert Andrew Wilson and Lucia Foglia. 2011. Embodied Cognition. In Stanford Encyclopedia of Philosophy. Stanford University Press.Google Scholar
- Jacob O. Wobbrock, Htet Htet Aung, Brandon Rothrock, and Brad A. Myers. 2005. Maximizing the guessability of symbolic input. CHI ’05 Extended Abstracts on Human Factors in Computing Systems. https://doi.org/10.1145/1056808.1057043Google ScholarDigital Library
- Jacob O Wobbrock, Meredith Ringel Morris, and Andrew D Wilson. 2009. User-defined gestures for surface computing. In Proceedings of the SIGCHI conference on human factors in computing systems. 1083–1092.Google ScholarDigital Library
Index Terms
- From Thoughts to Interaction: Designing Controls for Video Playback Gestures with Embodied Schemata
Recommendations
Embodied metaphors in tangible interaction design
For centuries, learning and development has been supported by physical activity and manipulating physical objects. With the introduction of embedded technologies, opportunities for employing tangible or embodied interaction for learning and development ...
Biting, Whirling, Crawling - Children's Embodied Interaction with Walk-through Displays
INTERACT '09: Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part IUnderstanding of embodied interaction in the context of walk-through displays and designing for it is very limited. This study examined children's intuitive embodied interaction with a large, semi-visible, projective walk-through display and space ...
Development of an eye-tracking pen display for analyzing embodied interaction
HI'11: Proceedings of the 2011 international conference on Human interface and the management of information - Volume Part IIn recent times, intuitive user interfaces such as the touch panel and pen display have become widely used in PCs and PDAs. Previously, the authors developed the bright pupil camera. They subsequently developed an eye-tracking pen display based on this ...
Comments