Abstract
A user experiment on multimodal interaction (speech, hand position and hand shapes) to study two major relationships: between the level of cognitive load experienced by users and the resulting multimodal interaction patterns; and how the semantics of the information being conveyed affected those patterns. We found that as cognitive load increases, users’ multimodal productions tend to become semantically more complementary and less redundant across modalities. This validates cognitive load theory as a theoretical background for understanding the occurrence of particular kinds of multimodal productions. Moreover, results indicate a significant relationship between the temporal multimodal integration pattern (7 patterns in this experiment) and the semantics of the command being issued by the user (4 types of commands), shedding new light on previous research findings that assign a unique temporal integration pattern to any given subject regardless of the communication taking place.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Baddeley, A.D.: Working Memory. Science 255, 5044, 556–559 (1992)
Bolt, R.A.: Put-That-There: Voice and Gesture at the Graphics Interface. In: 7th annual conference on Computer Graphics and Interactive Techniques, Seattle, Washington, United States, pp. 262–270. ACM Press, New York (1980)
Gupta, A.K., Anastasakos, T.: Dynamic Time Windows for Multimodal Input Fusion. In: Proc. 8th International Conference on Spoken Language Processing (INTERSPEECH 2004 - ICSLP), Jeju, Korea, October 4-8, 2004, pp. 1009–1012 (2004)
Hauptmann, A.G.: Speech and Gestures for Graphic Image Manipulation. In: CHI 1989, SIGCHI Conference on Human Factors in Computing Systems: Wings for the Mind, pp. 241–245. ACM Press, New York (1989)
Huang, X., Oviatt, S., Lunsford, R.: Combining user modeling and machine learning to predict users’ multimodal integration patterns. In: Renals, S., Bengio, S., Fiscus, J.G. (eds.) MLMI 2006. LNCS, vol. 4299, pp. 50–62. Springer, Heidelberg (2006)
Kettebekov, S.: Exploiting Prosodic Structuring of Coverbal Gesticulation. In: ICMI 2004: 6th international conference on Multimodal interfaces, State College, PA, USA, October 13-15, 2004, pp. 105–112. ACM Press, New York (2004)
Lisowska, A., Armstrong, S.: Multimodal input for meeting browsing and retrieval interfaces: preliminary findings. In: Renals, S., Bengio, S., Fiscus, J.G. (eds.) MLMI 2006. LNCS, vol. 4299, pp. 142–153. Springer, Heidelberg (2006)
Oviatt, S., Coulston, R., Lunsford, R.: When Do We Interact Multimodally? Cognitive Load and Multimodal Communication Pattern. In: ICMI 2004: 6th international conference on Multimodal interfaces, State College, PA, USA, October 13-15, 2004, pp. 129–136. ACM Press, New York (2004)
Oviatt, S., Coulston, R., Tomko, S., Xiao, B., Lunsford, R., Wesson, M., Carmichael, L.: Toward a Theory of Organized Multimodal Integration Patterns during Human-Computer Interaction. In: ICMI 2003, 5th international conference on Multimodal interfaces, Vancouver, British Columbia, Canada, November 05-07, 2003, pp. 44–51. ACM Press, New York (2003)
Oviatt, S., DeAngeli, A., Kuhn, K.: Integration and Synchronization of Input Modes During Multimodal Human-Computer Interaction. In: SIGCHI conference on Human factors in computing systems, Atlanta, Georgia, United States, March 22-27, 1997, pp. 415–422 (1997)
Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.M.: Cognitive Load Measurement as a Means to Advance Cognitive Load Theory. Educational psychologist 38(1), 63–71 (2003)
Quek, F., McNeill, D., Bryll, R., Kirbas, C., Arlsan, H., McCullough, K.E., Furuyama, N., Gesture, A.R.: Speech, and gaze cues for discourse segmentation. In: IEEE conference on computer vision and pattern recognition (CVPR 2000), Hilton head island, South Carolina, USA, June 13-15, 2000, pp. 247–254 (2000)
Ruiz, N., Taib, R., Chen, F.: Examining the redundancy of multimodal input. In: Proc. 20th annual conference of the Australian computer-human interaction special interest group (OzCHI 2006), Sydney, Australia, November 20-24, 2006, pp. 389–392 (2006)
Salber, D., Coutaz, J.: Applying the Wizard of Oz technique to the Study of Multimodal Systems. In: Bass, L.J., Unger, C., Gornostaev, J. (eds.) EWHCI 1993. LNCS, vol. 753, pp. 219–230. Springer, Heidelberg (1993)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Taib, R., Ruiz, N. (2008). Integrating Semantics into Multimodal Interaction Patterns. In: Popescu-Belis, A., Renals, S., Bourlard, H. (eds) Machine Learning for Multimodal Interaction. MLMI 2007. Lecture Notes in Computer Science, vol 4892. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-78155-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-540-78155-4_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-78154-7
Online ISBN: 978-3-540-78155-4
eBook Packages: Computer ScienceComputer Science (R0)