Summary
In this chapter we give an general overview of the modality fusion component of SmartKom. Based on a selection of prominent multimodal interaction patterns, we present our solution for synchronizing the different modes. Finally, we give, on an abstract level, a summary of our approach to modality fusion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
J. Alexandersson and N. Pfleger. Discourse Modeling, 2006. In this volume.
A. Berton, A. Kaltenmeier, U. Haiber, and O. Schreiner. Speech Recognition, 2006. In this volume.
R. Bolt. Put-That-There: Voice and Gesture at the Graphics Interface. Computer Graphics, 14(3):262–270, 1980.
C. Bregler, H. Hild, S. Manke, and A. Waibel. Improving Connected Letter Recognition by Lipreading. In: Proc. Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP-93), Minneapolis, MN, 1993.
B. Carpenter. The Logic of Typed Feature Structures. Cambridge University Press, Cambridge, UK, 1992.
P.R. Cohen, M. Johnston, D. McGee, S.L. Oviatt, J.A. Pittman, I. Smith, L. Chen, and J. Clow. QuickSet: Multimodal Interaction for Distributed Applications. In: Proc. 5th Int. Multimedia Conference (ACM Multimedia’ 97), pp. 31–40, Seattle, WA, 1997. ACM.
R. Engel. Natural Language Understanding, 2006. In this volume.
I. Gurevych, R. Porzel, and R. Malaka. Modeling Domain Knowledge: Know-How and Know-What, 2006. In this volume.
M. Johnston and S. Bangalore. Finite-State Methods for Multimodal Parsing and Integration. In: Finite-State Methods Workshop, ESSLLI Summer School on Logic Language and Information, Helsinki, Finland, August 2001.
M. Johnston, S. Bangalore, and G. Vasireddy. MATCH: Multimodal Access to City Help. In: Proc. ASRU 2001 Workshop, Madonna di Campiglio, Italy, 2001.
M. Johnston, P.R. Cohen, D. McGee, S.L. Oviatt, J.A. Pittman, and I. Smith. Unification Based Multimodal Integration. In: Proc. 35th ACL, pp. 281–288, Madrid, Spain, 1997.
M. Löckelt. Plan-Based Dialogue Management for Multiple Cooperating Applications, 2006. In this volume.
M. Minsky. A Framework for Representing Knowledge. In: P. Winston (ed.), The Psychology of Computer Vision, pp. 211–277, New York, 1975. McGraw-Hill.
S. Oviatt. Ten Myths of Multimodal Interaction. Communications of the ACM, 42(11):74–81, 1999.
S.L. Oviatt, A. DeAngeli, and K. Kuhn. Integration and Synchronization of Input Modes During Multimodal Human-Computer Interaction. In: Proc. CHI-97, pp. 415–422, 1997.
V. Pavlovic and T.S. Huang. Multimodal Tracking and Classification of Audio-Visual Features. In: AAAI Workshop on Representations for Multi-Modal Human-Computer Interaction, July 1998.
P. Poller and V. Tschernomas. Multimodal Fission and Media Design, 2006. In this volume.
R. Porzel, I. Gurevych, and R. Malaka. In Context: Integrating Domain-and Situation-Specific Knowledge, 2006. In this volume.
R.P. Shi, J. Adelhardt, A. Batliner, C. Frank, E. Nöth, V. Zeißler, and H. Niemann. The Gesture Interpretation Module, 2006. In this volume.
J. te Vrugt and T. Portele. Intention Recognition, 2006. In this volume.
W. Wahlster. User and Discourse Models for Multimodal Communication. In: J.W. Sullivan and S.W. Tyler (eds.), Intelligent User Interfaces, pp. 45–67, New York, 1991. ACM.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Engel, R., Pfleger, N. (2006). Modality Fusion. In: Wahlster, W. (eds) SmartKom: Foundations of Multimodal Dialogue Systems. Cognitive Technologies. Springer, Berlin, Heidelberg . https://doi.org/10.1007/3-540-36678-4_15
Download citation
DOI: https://doi.org/10.1007/3-540-36678-4_15
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23732-7
Online ISBN: 978-3-540-36678-2
eBook Packages: Computer ScienceComputer Science (R0)