EDWARD: full integration of language and action in a multimodal user interface

doi:10.1006/ijhc.1994.1022

International Journal of Human-Computer Studies

Volume 40, Issue 3, March 1994, Pages 473-495

https://doi.org/10.1006/ijhc.1994.1022 Get rights and content

Abstract

This paper presents EDWARD, a multimodal user interface fully integrating several interface styles, viz. natural language (Dutch), manipulation of graphical representations, menus, and command language. The focus is on the following two issues: (1) the new design principle of making all interface styles available at all times; and (2) the new generic approach in processing referring expressions, which is applied both in interpretation and generation, for all sorts of expressions, including multimodal deictic expressions such as "put that there" with simultaneous pointing gestures. EDWARD is a generic interface, currently applied to the file system domain.

References (0)

Cited by (8)

Multimodal analysis of speech and arm motion for prosody-driven synthesis of beat gestures
2016, Speech Communication
Citation Excerpt :
In this work, we construct a multimodal analysis framework to model the relationship between beat gestures, speech prosody and rhythm. Studies in diverse research areas suggest that human audio-visual communication is significantly rhythmic in nature, for example, in the way how spoken syllables and words are grouped together in time as in speech rhythm (Bolt, 1980; Ladd, 1996; Liberman, 1975) or how they are accompanied by body movements as in beat gestures (Bos et al., 1994; Tuite, 1993). In practice it is difficult to interpret the notion of rhythm for speech.
We propose a framework for joint analysis of speech prosody and arm motion towards automatic synthesis and realistic animation of beat gestures from speech prosody and rhythm. In the analysis stage, we first segment motion capture data and speech audio into gesture phrases and prosodic units via temporal clustering, and assign a class label to each resulting gesture phrase and prosodic unit. We then train a discrete hidden semi-Markov model (HSMM) over the segmented data, where gesture labels are hidden states with duration statistics and frame-level prosody labels are observations. The HSMM structure allows us to effectively map sequences of shorter duration prosodic units to longer duration gesture phrases. In the analysis stage, we also construct a gesture pool consisting of gesture phrases segmented from the available dataset, where each gesture phrase is associated with a class label and speech rhythm representation. In the synthesis stage, we use a modified Viterbi algorithm with a duration model, that decodes the optimal gesture label sequence with duration information over the HSMM, given a sequence of prosody labels. In the animation stage, the synthesized gesture label sequence with duration and speech rhythm information is mapped into a motion sequence by using a multiple objective unit selection algorithm. Our framework is tested using two multimodal datasets in speaker-dependent and independent settings. The resulting motion sequence when accompanied with the speech input yields natural-looking and plausible animations. We use objective evaluations to set parameters of the proposed prosody-driven gesture animation system, and subjective evaluations to assess quality of the resulting animations. The conducted subjective evaluations show that the difference between the proposed HSMM based synthesis and the motion capture synthesis is not statistically significant. Furthermore, the proposed HSMM based synthesis is evaluated significantly better than a baseline synthesis which animates random gestures based on only joint angle continuity.
An agent-based architecture for multimodal interaction
2001, International Journal of Human Computer Studies
In this paper, an executable generic process model is proposed for combined verbal and non-verbal communication processes and their interaction. The agent-based architecture can be used to create multimodal interaction. The generic process model has been designed, implemented and used to simulate different types of interaction between verbal and non-verbal communication processes: a non-verbal communication process can add and modify content to a verbal communication process, but can also provide a protocol for the (control of the) verbal communication process. With respect to the communication protocol both stimulus–response behaviour and deliberative behaviour have been modelled and simulated. The semantics of the model has been formalized by three-levelled partial temporal models, covering both the material and mental proceses and their relations.
The use and acquisition of artificial language: Some insights from psycholinguistics
1996, Computers in Human Behavior
Several observations of artificial language use are presented, suggesting an analogy in cognitive processing between artificial and natural language use. This article stresses the value psycholinguistics has for building a cognitive model of the artificial language user. An account for the presented observations of artificial language use is given in terms of psycholinguistics, for example, common ground, the principle of optimal design, referring as a collaborative process, generalization, and incremental parallel production. Implications of the analogy are also described and involve specific, testable predictions about user behavior, as well as tentative design recommendations for future artificial language interfaces that adapt to the cognitive processes underlying the user's artificial language use.
Language and gesture in interaction with the computer: Towards flexible dialogue
1999, Sistemi Intelligenti
Communicative rhythm in gesture and speech
1999, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Studies into full integration of language and action
1998, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

View all citing articles on Scopus

View full text

Regular ArticleEDWARD: full integration of language and action in a multimodal user interface

Abstract

Regular Article
EDWARD: full integration of language and action in a multimodal user interface