skip to main content
10.1145/3613905.3650874acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
Work in Progress

Mind the Mix: Exploring the Cognitive Underpinnings of Multimodal Interaction in Augmented Reality Systems

Published: 11 May 2024 Publication History

Abstract

Exploring the intricate dynamics of Multimodal Interaction (MMI) in Augmented Reality (AR), this study presents a novel conceptual framework, crafted from a review of cognitive theories. Our research delves into how input modalities, output modalities, and their combinations uniquely influence user experiences in AR environments. Recognizing a gap in the existing MMI literature, especially within the AR context, we propose a conceptual framework to understand these complex relationships. Our framework pinpoints three critical factors: the choice of input modality, the verbal processing code of outputs, and the synergistic effects of input-output combinations. These elements are hypothesized to significantly impact user interaction and performance in AR systems. This work-in-progress not only contributes to the theoretical discourse in HCI but also sets the stage for future empirical investigations, aiming to enhance user-centered design in the evolving field of AR technology.

Supplemental Material

MP4 File
Talk Video
Transcript for: Talk Video

References

[1]
A. Baddeley. 1992. Working memory. Science 255, 5044 (1992), 556-559.
[2]
A. Baddeley. 2000. The episodic buffer: a new component of working memory? Trends in cognitive sciences 4, 11 (2000), 417-423.
[3]
A. Baddeley. 2003. Working memory: looking back and looking forward. Nature reviews neuroscience 4, 10 (2003), 829-839.
[4]
A. D. Baddeley and G. Hitch. 1974. Working memory. In Psychology of learning and motivation, Vol. 8. Academic Press, 47-89.
[5]
M. D. Basil. 2012. Multiple resource theory. In Encyclopedia of the Sciences of Learning. 2384-2385.
[6]
M. Billinghurst, H. Kato, and S. Myojin. 2009. Advanced interaction techniques for augmented reality applications. In Proc. International Conference on Virtual and Mixed Reality. Springer, Berlin, Heidelberg, 13-22.
[7]
S. A. Brewster, P. C. Wright, and A. D. Edwards. 1994. The design and evaluation of an auditory-enhanced scrollbar. In Proc. SIGCHI Conference on Human Factors in Computing Systems. ACM, 173-179.
[8]
J. L. Burke 2006. Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. In Proc. 8th international conference on Multimodal interfaces. ACM, 108-117.
[9]
G. Calvert, C. Spence, and B. E. Stein (Eds.). 2004. The handbook of multisensory processes. MIT Press.
[10]
P. R. Cohen 1997. Quickset: Multimodal interaction for distributed applications. In Proc. fifth ACM international conference on Multimedia. ACM, 31-40.
[11]
E. Fintor, D. N. Stephan, and I. Koch. 2018. Emerging features of modality mappings in task switching: Modality compatibility requires variability at the level of both stimulus and response modality. Psychological Research 82, 1 (2018), 121-133.
[12]
N. Hollender 2010. Integrating cognitive load theory and concepts of human–computer interaction. Computers in human behavior 26, 6 (2010), 1278-1288.
[13]
W. Hürst and K. Vriens. 2016. Multimodal feedback for finger-based interaction in mobile augmented reality. In Proc. 18th ACM International Conference on Multimodal Interaction. ACM, 302-306.
[14]
S. Irawati 2006. An evaluation of an augmented reality multimodal interface using speech and paddle gestures. In International Conference on Artificial Reality and Telexistence. Springer, Berlin, Heidelberg, 272-283.
[15]
W. James. 1890. The Principles of Psychology. Henry Holt and Company, New York.
[16]
P. A. Kirschner, P. Ayres, and P. Chandler. 2011. Contemporary cognitive load theory research: The good, the bad and the ugly. Computers in Human Behavior 27, 1 (2011), 99-105.
[17]
M. J. Lazaro 2021. Interaction Modalities for Notification Signals in Augmented Reality. In Proc. 2021 International Conference on Multimodal Interaction. ACM, 470-477.
[18]
M. Lee 2013. A usability study of multimodal input in an augmented reality environment. Virtual Reality 17, 4 (2013), 293-305.
[19]
A. B. Naumann and I. Wechsung. 2008. Developing usability methods for multimodal systems: The use of subjective and objective measures. In Proc. International Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM), 8-12.
[20]
A. B. Naumann, I. Wechsung, and J. Hurtienne. 2010. Multimodal interaction: A suitable strategy for including older users? Interacting with Computers 22, 6 (2010), 465-474.
[21]
S. Oviatt and P. Cohen. 2022. The Paradigm Shift to Multimodality in Contemporary Computer Interfaces. Springer, Cham, Switzerland.
[22]
S. Oviatt. 1996. Multimodal interfaces for dynamic interactive maps. In Proc. SIGCHI conference on Human factors in computing systems. ACM, 95-102.
[23]
S. Oviatt 2000. Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions. Human-computer interaction 15, 4 (2000), 263-322.
[24]
A. Paivio. 1986. Dual coding and episodic memory: Subjective and objective sources of memory trace components. In Memory and cognitive capabilities: Symposium in memoriam of Hermann Ebbinghaus. North Holland, Amsterdam, 225-236.
[25]
A. Paivio. 1991. Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue canadienne de psychologie 45, 3 (1991), 255.
[26]
I. Rakkolainen 2021. Technologies for Multimodal Interaction in Extended Reality—A Scoping Review. Multimodal Technologies and Interaction 5, 12 (2021), 81.
[27]
S. Schaeffner, I. Koch, and A. M. Philipp. 2016. The role of sensory-motor modality compatibility in language processing. Psychological Research 80, 2 (2016), 212-223.
[28]
L. Schomaker. 1995. A taxonomy of multimodal interaction in the human information processing system.
[29]
W. Schnotz, M. Bannert, and T. Seufert. 2002. Toward an integrative view of text and picture comprehension: visualization effects on the construction of mental models. In The psychology of science text comprehension, 385-416.
[30]
J. Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257-285.
[31]
K. M. Stanney 2021. eXtended reality (XR) environments. In Handbook of human factors and ergonomics. 782-815.
[32]
E. Triantafyllidis, C. McGreavy, J. Gu, and Z. Li. 2020. Study of multimodal interfaces and the improvements on teleoperation. IEEE Access 8 (2020), 78213-78227.
[33]
A. Varghese. 2020. Exploring Bi-modal Feedback in Augmented Reality. In IndiaHCI'20: Proceedings of the 11th Indian Conference on Human-Computer Interaction. ACM, 55-61.
[34]
C. D. Wickens. 2002. Multiple resources and performance prediction. Theoretical issues in ergonomics science 3, 2 (2002), 159-177.
[35]
C. D. Wickens and C. M. Carswell. 2021. Information processing. In Handbook of human factors and ergonomics, 114-158.

Cited By

View all
  • (2025)Review of Multimodal Interaction in Optical See-Through Augmented RealityInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2442128(1-17)Online publication date: 17-Jan-2025

Index Terms

  1. Mind the Mix: Exploring the Cognitive Underpinnings of Multimodal Interaction in Augmented Reality Systems

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems
      May 2024
      4761 pages
      ISBN:9798400703317
      DOI:10.1145/3613905
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 11 May 2024

      Check for updates

      Author Tags

      1. Augmented Reality
      2. Cognitive Theories
      3. Input and Output Modalities
      4. Multimodal Interaction

      Qualifiers

      • Work in progress
      • Research
      • Refereed limited

      Conference

      CHI '24

      Acceptance Rates

      Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)275
      • Downloads (Last 6 weeks)41
      Reflects downloads up to 16 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Review of Multimodal Interaction in Optical See-Through Augmented RealityInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2442128(1-17)Online publication date: 17-Jan-2025

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      Full Text

      HTML Format

      View this article in HTML Format.

      HTML Format

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media