Skip to main content

Evaluation of Multimodal Behaviour of Embodied Agents

Cooperation between Speech and Gestures

  • Chapter
From Brows to Trust

Abstract

Individuality of Embodied Conversational Agents (ECAs) may depend on both the look of the agent and the way it combines different modalities such as speech and gesture. In this chapter, we describe a study in which male and female users had to listen to three short technical presentations made by ECAs. Three multimodal strategies of ECAs for using arm gestures with speech were compared: redundancy, complementarity, and speech-specialization. These strategies were randomly attributed to different-looking 2D ECAs, in order to test independently the effects of multimodal strategy and ECA’s appearance. The variables we examined were subjective impressions and recall performance. Multimodal strategies proved to influence subjective ratings of quality of explanation, in particular for male users. On the other hand, appearance affected likeability, but also recall performance. These results stress the importance of both multimodal strategy and appearance to ensure pleasantness and effectiveness of presentation ECAs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  • AAMAS (2002). Marriott, A., Pelachaud, C., Rist, T., Ruttkay, Zs., and Vilhjálmsson, H., editors. Proceedings of Workshop on Embodied Conversational Agents — Let’s Specify and Evaluate them!. AAMAS02, Bologna, Italy.

    Google Scholar 

  • AAMAS (2003). Pelachaud, C., Marriott, A., and Ruttkay, Zs., editors. Proceedings of Workshop on Embodied Conversational Characters as Individuals. AAMAS03, Melbourne, Australia.

    Google Scholar 

  • Abrilian, S., Buisine, S., Rendu, C., and Martin, J.-C. (2002). Specifying Cooperation between Modalities in Lifelike Animated Agents. In Proc. PRICAI02 Workshop on Lifelike Animated Agents: Tools, Functions, and Applications, pp. 3–8, Tokyo, Japan.

    Google Scholar 

  • André, E., Rist, T., Van Mulken, S., Klesen, M., and Baldes, S. (2000). The automated design of believable dialogues for animated presentation teams. In Cassell, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents. pp. 220–255, MIT Press, Cambridge.

    Google Scholar 

  • Ball, G. and Breese, J. (2000). Emotion and personality in a conversational character. In Cassell, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents. pp. 189–219, MIT Press, Cambridge.

    Google Scholar 

  • Cassell, J., Bickmore, T., Vilhjálmsson, H., and Yan, H. (2001). More than just a pretty face: Conversational protocols and the affordances of embodiment. Knowledge-Based Systems, 14: 55–64.

    Article  Google Scholar 

  • Cassell, J. and Stone, M. (1999). Living hand to mouth: Psychological theories about speech and gesture in interactive dialogue systems. In Proc. of AAAI99, pp. 34–42, North Falmouth, MA.

    Google Scholar 

  • Craig, S. D., Gholson, B., and Driscoll, D. (2002). Animated pedagogical agents in multimedia educational environments: Effects of agent properties, picture features, and redundancy. Journal of Educational Psychology, 94: 428–434.

    Article  Google Scholar 

  • Feldman, R. S., Philippot, P., and Custrini, R. J. (1991). Social competence and nonverbal behavior. In Feldman, R.S. and Rimé, B., editors, Fundamentals of Nonverbal Behavior. pp. 329–350, Cambridge University Press, Cambridge, England.

    Google Scholar 

  • Giese, M.A. and Poggio, T. (2003). Neural mechanisms for recognition of biological movements. Nature Reviews, 4: 179–192.

    Article  Google Scholar 

  • Goldin-Meadow, S. (1999a). The role of gesture in communication and thinking. Trends in Cognitive Sciences, 3(11): 419–429.

    Article  Google Scholar 

  • Goldin-Meadow, S., Kim, S., and Singer, M. (1999b). What the teacher’s hands tell the student’s mind about math. Journal of Educational Psychology, 91: 720–730.

    Article  Google Scholar 

  • Kendon, A. (1980). Gesticulation and speech: two aspects of the process of utterance. In Key, M.R., editor, The relationship of Verbal and Nonverbal Communication. pp. 207–228, Mouton Publishers, The Hague, The Netherlands.

    Google Scholar 

  • Kimura, D. (1999). Sex and Cognition. MIT Press, Cambridge.

    Google Scholar 

  • Kohar, H. and Ginn, I. (1997). Mediators: Guides through online TV services. In Electronic Proc. CHI97. URL: http://www.acm.org/sigchi/chi97/proceedings/demo/hk.htm/

  • Martin, J. C., Grimard, S., and Alexandri, K. (2001). On the annotation of the multimodal behavior and computation of cooperation between modalities. In Proc. AAMAS01 Workshop on Representing, Annotating, and Evaluating Non-Verbal and Verbal Communicative Acts to Achieve Contextual Embodied Agents, pp. 1–7, Montreal, Canada.

    Google Scholar 

  • McBreen, H., Anderson, J., and Jack, M. (2001). Evaluating 3D embodied conversational agents in contrasting VRML retail applications. In Proc. Int. Conf. on Autonomous Agents Workshop on Multimodal Communication and Context in Embodied Agents, pp. 83–87, Montreal, Canada.

    Google Scholar 

  • McNeill, D. (1987). Psycholinguistics: A new approach. Harper and Row, New York.

    Google Scholar 

  • McNeill, D. (1992). Hand and Mind, University of Chicago Press, Chicago, IL.

    Google Scholar 

  • Moreno, K. N., Klettke, B., Nibbaragandla, K., and Graesser, A. C. (2002). Perceived characteristics and pedagogical effcacy of animated conversational agents. In Proc. of ITS02, pp. 963–971, Biarritz, France and San Sebastian, Spain.

    Google Scholar 

  • Moreno, R. and Mayer, R. E. (2002). Verbal redundancy in multimedia learning: when reading helps listening. Journal of Educational Psychology, 94: 156–163.

    Article  Google Scholar 

  • van Mulken, S. André, E., and Müller, J. (1998). The Persona effect: How substantial is it? In Proc. of HCI98, pp. 53–66, Berlin, Germany.

    Google Scholar 

  • Myers, J.L. (1979). Fundamentals of Experimental Design. Third Edition, Allyn and Bacon Inc, Boston.

    Google Scholar 

  • Nijholt, A. (2001). Towards multi-modal emotion display in embodied agents. In Proc. of Artificial Neural Networks and Expert Systems 2001, pp. 229–231, Dunedin, New Zealand.

    Google Scholar 

  • Pelachaud, C., Carofiglio, V., De Carolis, B., De Rosis, F., and Poggi, I. (2002). Embodied contextual agent in information delivering application. In Proc. of AAMAS02, pp. 758–765, Bologna, Italy.

    Google Scholar 

  • Rickel, J. and Johnson, W. L. (1999). Animated agents for procedural training in virtual reality: Perception, cognition, and motor control. Applied Artificial Intelligence, 13: 343–382.

    Article  Google Scholar 

  • Rimé, B. and Schiaratura, L. (1991). Gesture and speech. In: Feldman, R.S. and Rimé, B., editors. Fundamentals of Nonverbal Behavior. pp. 239–284, Cambridge University Press, Cambridge, England.

    Google Scholar 

Download references

Authors

Editor information

Zsófia Ruttkay Catherine Pelachaud

Rights and permissions

Reprints and permissions

Copyright information

© 2004 Kluwer Academic Publishers

About this chapter

Cite this chapter

Buisine, S., Abrilian, S., Martin, JC. (2004). Evaluation of Multimodal Behaviour of Embodied Agents. In: Ruttkay, Z., Pelachaud, C. (eds) From Brows to Trust. Human-Computer Interaction Series, vol 7. Springer, Dordrecht. https://doi.org/10.1007/1-4020-2730-3_8

Download citation

  • DOI: https://doi.org/10.1007/1-4020-2730-3_8

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-1-4020-2729-1

  • Online ISBN: 978-1-4020-2730-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics