Skip to main content
Springer Nature Link
Log in

MARC: a framework that features emotion models for facial animation during human–computer interaction

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Emotions and their expressions by virtual characters will play a key role in future affective human–machine interfaces. Recent advances in the psychology of emotions and recent progress in computer graphics allow researchers to animate virtual characters that are capable of subtly expressing emotions. Existing virtual agent systems are nevertheless often limited in terms of underlying emotional models, visual realism, and real-time interaction capabilities. In this paper, we explain how we explored four computational models of emotions that are oriented towards interactive facial animation. The models that we implemented correspond to different approaches to emotions: a categorical approach, a dimensional approach, a cognitive approach, and a social approach. We explain how we implemented and evaluated these models in our Multimodal Affective and Reactive Character framework. MARC is designed for animating interactive expressive virtual agents with different levels of interactivity. The advantages, drawbacks and complementarity of these approaches are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. André E, Klesen M, Gebhard P, Allen S, Rist T (2000) Integrating models of personality and emotions into lifelike characters. In: Affective interactions, pp 150–165

  2. Becker C, Kopp S, Wachsmuth I (2007) Why emotions should be integrated into conversational agents. In: Nishida T (ed) Engineering approaches to conversational informatics. Wiley, Chichester, pp 49–68

    Chapter  Google Scholar 

  3. Becker-Asano C, Wachsmuth I (2008) Affect simulation with primary and secondary emotions. In: Intelligent virtual agent. Springer, Berlin Heidelberg, pp 15–28

  4. Becker-Asano C, Wachsmuth I (2010) Affective computing with primary and secondary emotions in a virtual human. Auton Agents Multi-Agent Syst 20(1):32–49

    Article  Google Scholar 

  5. Bickmore T, Pfeifer L, Schulman D (2011) Relational agents improve engagement and learning in science museum visitors. Intelligent Virtual Agents, Reyjavik, Iceland, pp 55–67

  6. Broekens J (2012) In defense of dominance: PAD usage in computational representations of affect. Int J Synth Emot 3(1):33–42

    Article  Google Scholar 

  7. Cassell J, Vilhjàlmsson H, Bickmore T (2001) BEAT: the behavior expression animation toolkit. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques, pp. 477–486

  8. Clavel C, Martin JC (2009) PERMUTATION: a corpus-based approach for modeling personality and multimodal expression of affects in virtual characters. In: Digital human modeling Springer, Berlin Heidelberg, pp 211–220

  9. Courgeon M, Buisine S, Martin J-C (2009) Impact of expressive wrinkles on perception of a virtual character’s facial expressions of emotions. In: Proceedings of the 9th international conference on intelligent virtual agents (IVA 09). Amsterdam, The Netherlands, 10–12 September 2009, pp 201–214

  10. Courgeon M, Clavel C, Martin J-C (2009) Appraising Emotional Events during a real-time interactive game. In: Proceedings of the ICMI 2009 workshop on affective computing (AFFINE), Cambrigde, USA, 1–6 November 2009

  11. Courgeon M, Amorim M-A, Giroux C, Martin J-C (2010) Do users anticipate emotion dynamics in facial expressions of a virtual character? In: Proceedings of the 23rd international conference on computer animation and social agents (CASA 2010), Saint Malo, France, 31 May–2 June 2010

  12. Courgeon M, Clavel C, Tan N, Martin JC (2011) Front view vs. side view of facial and postural expressions of emotions in a virtual character. In: Transactions on edutainment. Springer, Berlin Heidelberg, pp 132–143

  13. Courgeon M, Martin J-C, Jacquemin C (2008) User’s gestural exploration of different virtual agents’ expressive profiles. In: Proceedings of 7th international conference on autonomous agents and multiagent systems (AAMAS 08), vol 3. Estoril, Portugal, 12–16 May 2008, pp 1237–1240

  14. de Gelder B, Van den Stock J (2011) Real faces, real emotions: perceiving facial expressions in naturalistic contexts of voices, bodies and scenes. In: Calder GRAJ, Haxby JV, Johnson MH (eds) The handbook of face perception. Oxford University Press, Oxford, pp 535–550

  15. Dias J, Païva A (2005) Feeling and reasoning: a computational model for emotional characters. In: Progress in artificial intelligence. Springer, Berlin Heidelberg, pp 127–140

  16. Ekman P (1982) Methods for measuring facial actions. In: Scherer K, Ekman P (ed) Handbook on methods of nonverbal communication research. Cambridge University Press, New York

  17. Ekman P, Friesen W (1975) Unmasking the face. A guide to recognizing facial clues. Prentice-Hall Inc., Englewood Cliffs, New Jersey

  18. Ferreira N, Mascarenhas S, Paiva A, Dignum F, McBreen J, Degens N, Jan Hofstede G (2012) Generating norm-related emotions in virtual agents. In: International conference on intelligent virtual agents, vol 7502, pp 97–104

  19. Gross J, Feldman Barrett L (2011) Emotion generation and emotion regulation: one or two depends on your point of view. Emot Rev 3(1):8–16

    Article  Google Scholar 

  20. Golan O, Baron-Cohen S, Hill J (2006) The Cambridge mindreading (CAM) face-voice battery: testing complex emotion recognition in adults with and without asperger syndrome. J Autism Dev Disord 36(2): 169–183

    Google Scholar 

  21. Gunes H, Pantic M (2010) Automatic, dimensional and continuous emotion recognition. Int J Synth Emot 1(1):68–99

    Article  Google Scholar 

  22. Gunes H, Piccardi M, Pantic M (2008) From the lab to the real world: affect recognition using multiple cues and modalities. In: Or J (ed) Affective computing: focus on emotion expression, synthesis, and recognition. I-Tech Education and Publishing, Vienna, Austria, pp 185–218

  23. Ho WC, Watson S (2006) Autobiographic knowledge for believable virtual characters. In: Intelligent virtual agents. Springer, Berlin Heidelberg, pp 383–394

  24. Hudlicka E, Gunes H (2012) Benefits and limitations of continuous representations of emotions in affective computing: introduction to the special issue (PDF). Int J Syn Emot 3(1):i–vi. http://www.igi-global.com/Files/Ancillary/64d30392-4125-4708-9e7f-9599f99a6657_IJSE3%281%29Preface.pdf

  25. Kensinger EA, Corkin S (2004) Two routes to emotional memory: distinct neural processes for valence and arousal. Proc Natl Acad Sci USA. 101(9): 3310–3315

    Google Scholar 

  26. Knapp RB, Kim J, André E (2011) Physiological signals and their use in augmenting emotion recognition for human–machine interaction. In: Emotion-oriented systems. Springer, Berlin Heidelberg, pp 133–159

  27. Kopp S, Stocksmeier T, Gibbon D (2007) Incremental multimodal feedback for conversational agents. In: Intelligent virtual agents. Springer, Berlin Heidelberg, pp 139–146

  28. Leite I, Castellano G, Pereira A, Martinho C, Paiva A, McOwan PW (2009) Designing a game companion for long-term social interaction. In: Proceedings of the international workshop on affective aware virtual agents and social robots. ACM, p 10

  29. Manstead AS, Fischer AH (2001) Social appraisal: the social world as object of and influence on appraisal processes. In: Dans K, Scherer R, Schorr A, Johnstone T (eds) Appraisal processes in emotion: theory, methods, research, pp 221–232

  30. Marsella S (2010) Modeling emotion and its expression in virtual humans. Intell Tutor Syst 1(2):1–2

    Google Scholar 

  31. Marsella S, Gratch J (2006) EMA: a computational model of appraisal dynamics. In: European meeting on cybernetics and systems research

  32. Martin P, Bourdot P, Touraine D (2011) A reconfigurable architecture for multimodal and collaborative interactions in virtual environments (technote). In 3DUI, 11: Proceedings of IEEE symposium on 3D User Interfaces (3DUI). IEEE Computer Society

  33. Mumenthaler C, Sander D (2010) Social appraisal, how the evaluation of others influences our own perception of emotional facial expressions. In: XVII Annual cognitive neuroscience society meeting, pp 17–20

  34. Niewiadomski R, Bevacqua E, Mancini M, Pelachaud C (2009) Greta: an interactive expressive ECA system. In: Proceedings of the 8th international conference on autonomous agents and multiagent systems, pp 1399–1400

  35. Ochs M, Niewiadomski R, Pelachaud C, Sadek D (2005) Intelligent expressions of emotions. In: Affective computing and intelligent interaction. Springer, Berlin Heidelberg, pp 707–714

  36. Picard RW (1997) Affective computing. MIT Press, Cambridge

  37. Rivière J, Adam C, Pesty S, Nakano Y, Neff M, Paiva A, Walker M (2012) A reasoning module to select ECA’s communicative intention. In: Proceedings of the 12th international conference on intelligent virtual agents (IVA’12), vol. 7502. Lecture notes in computer science, Heidelberg, pp 447–454

  38. Riviere J, Adam C, Pesty S, Pelachaud C, Guiraud N, Longin D, Lorini E (2011) Expressive multimodal conversational acts for SAIBA agent. In: Intelligent virtual agents, pp 316–323

  39. Russell JA, Mehrabian A (1977) Evidence for a three-factor theory of emotions. Res Pers 11(3):273–294

    Article  Google Scholar 

  40. Russell J (1994) Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychol Bull 115(1):102–114

    Article  Google Scholar 

  41. Russell J (2012) Introduction to special section: on defining emotion. Emot Rev 4(4):337–338

    Article  Google Scholar 

  42. Ruttkay S, Noot H, Hagen P (2003) Emotion disc and emotion squares: tools to explore the facial expression face. Comput Graph Forum 22(1):49–53

    Google Scholar 

  43. Scherer KR (1984) On the nature and function of emotion: a component process approach. In: Approaches to emotion, pp 293–317

  44. Scherer KR (2001) Appraisals considered as a process of multilevel sequential checking. In: Appraisal process in emotion: theory, methods, research, pp 92–120

  45. Thornton I (1998) The perception of dynamic human faces. PhD Thesis, University of Oregon

  46. Tomkins SS (1984) Affect theory. In: Scherer DK, Ekman P (eds) Approaches to emotion. Erlbaum, Hillsdale NJ, pp 163–195

  47. Wallraven C, Breidt M, Cunningham D, Bülthoff HH (2005) Psychophysical evaluation of animated facial expressions. In: 2nd Symposium on applied perception in graphics and visualization. ACM Press, New York, pp 17–24

Download references

Author information

Authors and Affiliations

  1. Laboratoire d’Informatique pour la Mécanique et les Science de l’Ingénieur, Bâtiment 508, 91403 , Orsay Cedex, France

    M. Courgeon & C. Clavel

  2. Université Paris Sud, 15 rue Georges Clémenceau, 91405,  Orsay Cedex, France

    C. Clavel

Authors
  1. M. Courgeon
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. C. Clavel
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to M. Courgeon.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Courgeon, M., Clavel, C. MARC: a framework that features emotion models for facial animation during human–computer interaction. J Multimodal User Interfaces 7, 311–319 (2013). https://doi.org/10.1007/s12193-013-0124-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-013-0124-1

Keywords