Abstract
During face-to-face communication, the dialog partners can see and hear each other. Each speaker produces a variety of phenomena parallel to speech. Some of them, e.g. intonation, are coded vocally, others are coded by motor responses (facial expression, gestures, etc.). If human-computer-interaction (HCI) tries to mimic this situation, at least some non-verbal phenomena have to be integrated into natural language input and output. A multitude of new devices (mouse, joystick, touch-screens, etc.) have enabled this transition to multimodal HCI. Gestures which illustrate the content of the verbal message are especially suitable for integration into HCI. A relevant subset of them is pointing gestures, which specify elements of the visual context. They are performed frequently because their use shortens and simplifies the verbal output. As an illustration of these considerations, the NL dialog system XTRA (University of Saarbrücken) is presented. It allows reference to elements of a tax form by the combination of textual input and simulated pointing gestures. In order to explore the regularities of this “form deixis,” an experiment has been carried out within the framework of the XTRA-project. Furthermore, its results were taken for an evaluation of the currently used simulation technique.
Michael Wille is a researcher in the AI laboratory at the University of Saarbrücken. He has studied computer science, economics and cognitive psychology. he has worked on expert systems for SIEMENS (hardware diagnosis). His master's thesis (1989) was called “Evaluation and Extension of a Module for the Simulation and Analysis of Pointing Gestures.” His main research interest is multimedia interaction.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Allgayer, J. “Eine Graphikkomponente zur Integration von Zeigehandlungen in natürlichsprachliche KI-Systeme.” Processings der 16. GI-Jahrestagung. Berlin: Springer, 1986, pp. 284–98.
Allgayer, J., K. Harbusch, A. Kobsa, C. Reddig, N. Reithinger, D. Schmauks. “XTRA: A Natural-Language Access System to Expert Systems.” International Journal of Man-Machine Studies (1989). To Appear.
Buxton, W. and B. A. Myers. “A Study in Two-Handed Input.” Proceedings CHI 86 Human Factors in Computing Systems. New York: ACM. 1986, pp. 321–26.
Ekman, P. and W. V. Friesen. “The Repertoire of Nonverbal Behavior: Categories, Origins, and Coding.” Semiotica, 1, (1969), pp. 49–98.
Hinrichs, E. and L. Polanyi. “Pointing The Way: A Unified Treatment Of Referential Gesture In Interactive Discourse.” Paper from Parasession on Pragmatics and Grammatical Theory at the 22nd Regional Meeting, Chicago Linguistic Society, Chicago, 1987, pp. 298–314.
Kobsa, A., J. Allgayer, C. Reddig, N. Reithinger, D. Schmauks, K. Harbusch and W. Wahlster. “Combining Deictic Gestures and Natural Language for Referent Identification.” Proceedings of the 11th International Conference on Computational Linguistics, Bonn, Germany, 1986, pp. 356–61.
McKeown, K. R.Text Generation. Cambridge University Press, 1985.
Minsky, M. “Manipulating Simulated Objects with Real-World Gestures Using a Force and Position Sensitive Screen.” Computer Graphics 18, 3 (1984) 195–203.
Neal, J. G. and S. C. Shapiro. “Intelligent Multi-Media Interface Technology.” In Architectures for Intelligent Interfaces: Elements and Prototypes. Ed. Sullivan, J. W. and S. W. Tyler. Reading: Addison Wesley, 1988.
Pickering, J. A. “Touch-sensitive Screens: The Technologies and their Application.” International Journal of ManMachine Studies, 25 (1986), 249–69.
Reithinger, N. “Generating Referring Expressions and Pointing Gestures.” In Natural Language Generation. Ed. G. Kempen. Dordrecht: Kluwer, 1987a, pp. 71–81.
Reithinger, N. “Ein erster Blick auf POPEL — Wie wird was gesagt?” In Proceedings of GWAI-87. Ed. K. Morik. Berlin: Springer, pp. 315–19.
Reithinger, N. “POPEL — A Parallel and Incremental Natural Language Generation System.” Paper presented at the 4th International Workshop on Natural Language Generation, Santa Catalina Island, 1988.
Scherer, K. R. “Die Funktionen des nonverbalen Verhaltens im Gespräch.” In Nonverbale Kommunikation. Ed. Scherer, K. R. and H. G. Wallbott, Hrsg. Weinheim: Beltz, 1979.
Schmauks, D. “Formulardeixis and ihre Simulation auf dem Bildschirm. Ein Überblick aus linguistischer Sicht.” Memo Nr. 4, SFB 314, FB Informatik, Universität Saarbrücken. Also in Conceptus, 55 (1988),83–102.
Schmauks, D. “Form and Funktion von Zeigegesten. Ein interdisziplinärer Uberblick.” Bericht Nr. 10, SFB 314, FB Informatik, Universität Saarbrücken, 1986b.
Schmauks, D. “Natural and Simulated Pointing.” Proceedings of the 3rd European ACL Conference, Kopenhagen, Danmark, pp. 179–85. Also: Bericht Nr. 16, SFB 314, FB Informatik, Universität Saarbrücken.
Schmauks, D. and N. Reithinger. “Generating Multimodal Output — Conditions, Advantages and Problems.” Proceedings of the 12th International Conference on Computational Linguistics, Budapest, Hungary, 1988, pp. 584–88. Also: Bericht Nr. 29, SFB 314, FB Informatik, Universität Saarbrücken.
Wahlster, W. “User and Discourse Models for Multimodal Communication.” In Architectures for Intelligent Interfaces: Elements and Prototypes. Ed. Sullivan, J. W. and S. W. Tyler. Reading: Addison Wesley, 1988.
Wetzel, R. P., K. H. Hanne and J. P. Hoepelmann. “DIS-QUE: Deictic Interaction System-Query Environment.” LOKI Report KR-GR 5.3/KR-NL 5, FhG, IAO, Stuttgart, 1987.
Wille, M. Evaluation and Ausbau einer Analysekomponente für Zeigegesten. Diplomarbeit, FB Informatik, Universität des Saarlandes, 1989.
Zimmerman, T. G., J. Lanier, C. Blanchard, S. Bryson and Y. Harvill. “A Hand Gesture Interface Device.” Proceedings CHT'87 Human Factors in Computing Systems. New York: ACM, 1987, pp. 189–92.
Author information
Authors and Affiliations
Additional information
Dagmar Schmauks is a researcher in the AI laboratory at the University of Saarbrücken. She has pursued studies in philosophy, linguistics and psychology. Her doctoral thesis (1990) was entitled “Deixis in Man-Machine-interaction.” Since 1985 she has worked in a special collaborative program on AI and knowledge-based systems at the University pf Saarbrücken. Her main research interest is multimedia interaction.
Rights and permissions
About this article
Cite this article
Schmauks, D., Wille, M. Integration of communicative hand movements into human-computer-interaction. Comput Hum 25, 129–140 (1991). https://doi.org/10.1007/BF00124149
Issue Date:
DOI: https://doi.org/10.1007/BF00124149