Instructing animated agents: Viewing language in behavioral terms

Abstract

One activity of Penn's Center for Human Modelling and Simulation has been the exploration of natural language instructions and other high-level task specifications to create animated simulations of virtual human agents carrying out tasks. The work builds on JACK, an animation system developed at Penn, that provides simulated human models with a growing repertoire of naturalistic behaviors. The value in using high-level task specifications to create animated simulations is that the same specification can be used to produce different animations in different situations, without additional animator or programmer intervention.

But animated simulation driven by natural language instructions can provide another benefit, by forcing us to consider what aspects of language convey information relevant to behavior. What our studies to date have revealed is that more of an utterance conveys such information than its main verb and argument structure.

To demonstrate an analysis of linguistic constructs in terms of behavioral specifications and constraints, I show how instructions containing ‘until’ clauses can be analysed in terms of perceptual activities and the conditions they are used to assess, and how the resulting analysis contributes to understanding how an agent is supposed to carry these instructions out.