Skip to main content
SpringerLink
Log in

Animated Pedagogical Agents Performing Affective Gestures Extracted from the GEMEP Dataset: Can People Recognize Their Emotions?

  • Conference paper
  • First Online:
ArtsIT, Interactivity and Game Creation (ArtsIT 2023)

Included in the following conference series:

  • 57 Accesses

Abstract

The study reported in the paper focused on applying a set of affective body gestures extracted from the Geneva Multimodal Emotion Portrayals (GEMEP) dataset to two pedagogical animated agents in an online lecture context and studying the effects of those gestures on subjects’ perception of the agents’ emotions. 131 participants completed an online survey where they watched different animations featuring a female and a male animated agent expressing six emotions (anger, joy, sadness, disgust, fear, and surprise) while delivering a lecture segment. After watching the animations, subjects were asked to identify the agents’ emotions. Findings showed that only one expression of the angry emotion by the female agent was recognized with an acceptable level of accuracy (recognition rate >75%), while the remaining emotions showed low recognition rates ranging from 1.5% to 64%. A mapping of the results on Russel’s Circumplex model of emotion showed that participants’ identification of levels of emotion arousal and valence was slightly more accurate than recognition of emotion quality but still low (recognition rates <75% for 5 out of 6 emotions). Results suggest that hand and arm gestures alone are not sufficient for conveying the agent’s emotion type and the levels of emotion valence and arousal.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adamo, N., Mousas, C., Mayer, R.: NSF Award # 2201019 - Collaborative Research: Using Artificial Intelligence to Transform Online Video Lectures into Effective and Inclusive Agent-Based Presentations (2022). https://www.nsf.gov/awardsearch/showAward?AWD_ID=2201019&HistoricalAwards=false&_ga=2.46658222.863696423.1696858525-1636934363.1694122541

  2. Annetta, L.A., Holmes, S.: Creating presence and community in a synchronous virtual learning environment using avatars. Intern. J. Instruct. Technol. Dist. Learn. 3, 27–43 (2006)

    Google Scholar 

  3. Atkinson, A.P., Dittrich, W.H., Gemmell, A.J., Young, A.W.: Emotion perception from dynamic and static body expressions in point-light and full-light displays. Perception 33(6), 717–746 (2004). https://doi.org/10.1068/p5096

    Article  Google Scholar 

  4. Bänziger, T., Scherer, K.R.: Introducing the Geneva Multimodal Emotion Portrayal (GEMEP) corpus. In: Scherer, K.R., Bänziger, T., Roesch, E.B. (eds.) Blueprint for affective computing: A sourcebook, pp. 271–294. Oxford university Press, Oxford, England (2010)

    Google Scholar 

  5. Barrett, L.F., Russell, J.A. (eds.): The Psychological Construction of Emotion. The Guilford Press (2015)

    Google Scholar 

  6. Baylor, A.L., Kim, S.: Designing nonverbal communication for pedagogical agents: when less is more. Comput. Hum. Behav.. Hum. Behav. 25(2), 450–457 (2009)

    Article  Google Scholar 

  7. Blythe, E., Garrido, L., Longo, M.R.: Emotion is perceived accurately from isolated body parts, especially hands. Cognition 230, 105260 (2023). https://doi.org/10.1016/j.cognition.2022.105260. Epub 2022 Sep 1. PMID: 36058103

  8. Cheng, J., Zhou, W., Lei, X., Adamo, N., Benes, B.: The Effects of Body Gestures and Gender on Viewer’s Perception of Animated Pedagogical Agent’s Emotions. In: Kurosu, M. (ed.) Human-Computer Interaction. Multimodal and Natural Interaction: Thematic Area, HCI 2020, Held as Part of the 22nd International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part II, pp. 169–186. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-49062-1_11

    Chapter  Google Scholar 

  9. Cui, J., Popescu, V., Adamo-Villani, N., Cook, S.W., Duggan, K.A., Friedman, H.S.: Animation stimuli system for research on instructor gestures in education. IEEE Comput. Graph. Appl.Comput. Graph. Appl. 37(4), 72–83 (2017). https://doi.org/10.1109/MCG.2017.3271471

    Article  Google Scholar 

  10. Ekman, P.:. Emotions revealed. New York/London: Times Books (US)/Weidenfeld & Nicolson (2003)

    Google Scholar 

  11. Ekman, P., Friesen, W.: The repertoire of nonverbal behavior: categories, origins, usage, and coding. Semiotica 1(1), 49–98 (1969)

    Article  Google Scholar 

  12. Ennis, C., Hoyet, L., Egges, A., McDonnell, R.: Emotion capture: emotionally expressive characters for games. In: Proceedings of Motion on Games (2013). https://doi.org/10.1145/2522628.2522633

  13. Fiorella, L., Mayer, R.E.: Principles based on social cues in multimedia learning: Personalization, voice, embodiment, and image principles. In: Mayer, R.E., Fiorella, L. (eds.) The Cambridge Handbook of Multimedia Learning, 3rd edn, pp. 277–286. Cambridge University Press, New York (in press)

    Google Scholar 

  14. Guo, Y.R., Goh, D.H.-L.: Affect in embodied pedagogical agents: meta- analytic review. J. Educ. Comput. Res. 53(1), 124–149 (2015). https://doi.org/10.1177/0735633115588774

    Article  Google Scholar 

  15. Johnson, W.L., Lester, J.C.: Face-to-face interaction with pedagogical agents, twenty years later. Int. J. Artif. Intell. Educ.Artif. Intell. Educ. 26(1), 25–36 (2016)

    Article  Google Scholar 

  16. Karg, M., Samadani, A.A., Gorbet, R., Kühnlenz, K., Hoey, J., Kulić, D.: Body movements for affective expression: a survey of automatic recognition and generation. IEEE Trans. Affect. Comput.Comput. 4(4), 341–359 (2013)

    Article  Google Scholar 

  17. Larsson, P.: Discerning emotion through movement – a study of body language in portraying emotion in animation, pp. 6–7 (2014) MS Thesis. Retrieved from: http://www.divaportal.org/smash/record.jsf?pid=diva2%3A723103&dswid=4060

  18. Lawson, A.P., Mayer, R.E., Adamo-Villani, N., Benes, B., Lei, X., Cheng, J.: Do learners recognize and relate to the emotions displayed by virtual instructors? Int. J. Artif. Intell. Educ.Artif. Intell. Educ. 31, 134–153 (2021)

    Article  Google Scholar 

  19. Lhommet, M., Marsella, S.: Expressing emotion through posture and gesture. In: Calvo, R., D’Mello, S., Gratch, J., Kappas, A. (eds.) The Oxford Handbook of Affective Computing, pp. 273–285. Oxford University Press, Oxford (2015)

    Google Scholar 

  20. Martha, A.S., Santoso, H.B.: The design and impact of the pedagogical agent: a systematic literature review. J. Educ. Online, 16(1) (2019)

    Google Scholar 

  21. Mayer, R.E.: Multimedia learning, 3rd edn. Cambridge University Press, New York (2021)

    Google Scholar 

  22. Mayer, R.E., Fiorella, L., Stull, A.: Five ways to increase the effectiveness of instructional video. Educ. Tech. Res. Dev. 68, 837–852 (2020)

    Article  Google Scholar 

  23. Mayer, R.E., DaPra, C.S.: An Embodiment effect in computer-based learning with animated pedagogical agents. J. Exp. Psychol. Appl. 18(3), 239–252 (2012)

    Article  Google Scholar 

  24. Mayer, R.E.: Searching for the role of emotions in e-learning. Learn. Instr. 70, 101213 (2020). https://doi.org/10.1007/978-3-030-90436-4_38

    Article  Google Scholar 

  25. Poggiali, J.: Student responses to an animated character in information literacy instruction. Library Hi Tech 36(1), 29–42 (2018). https://doi.org/10.1108/LHT-12-2016-0149

    Article  Google Scholar 

  26. Posner, J., Russell, J.A., Peterson, B.S. (2005). The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev Psychopathol. 2005 Summer 17(3), 715–734 PMID: 16262989; PMCID: https://doi.org/10.1017/S0954579405050340

  27. Rosenberg-Kima, R.B., Baylor, A.L., Plant, E.A., Doerr, C.E.: Interface agents as social models for female students: the effects of agent visual presence and appearance on female students’ attitudes and beliefs. Comput. Hum. Behav.. Hum. Behav. 24, 2741–2756 (2008)

    Article  Google Scholar 

  28. Ross, P., Flack, T.: Removing hand form information specifically impairs emotion recognition for fearful and angry body stimuli. Perception 49(1), 98–112 (2020). https://doi.org/10.1177/0301006619893229

    Article  Google Scholar 

  29. Russell, J.A.: Core affect and the psychological construct of emotion. Psychol. Rev. 110, 145–172 (2003)

    Article  Google Scholar 

  30. Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)

    Article  Google Scholar 

  31. Sawada, M., Suda, K., Ishii, M.: Expression of emotions in dance: relation between arm movement characteristics and emotion. Percept. Mot. Skills 97, 697–708 (2003)

    Article  Google Scholar 

  32. Schroeder, N.L., Adesope, O.O., Gilbert, R.B.: How effective are pedagogical agents for learning? a meta-analytic review. J. Educ. Comput. Res. 49(1), 1–39 (2013)

    Article  Google Scholar 

  33. Wang, N., Johnson, W.L., Mayer, R.E., Rizzo, P., Shaw, E., Collins, H.: The politeness effect: pedagogical agents and learning outcomes. Int. J. Hum. Comput. Stud.Comput. Stud. 66, 98–112 (2008)

    Article  Google Scholar 

Download references

Acknowledgments

This research is funded by NSF award# 2201019 – “Collaborative Research: Using Artificial Intelligence to Transform Online Video Lectures into Effective and Inclusive Agent-Based Presentations”.

Author information

Authors and Affiliations

  1. Purdue University, West Lafayette, IN, 47907, USA

    Magzhan Mukanova, Nicoletta Adamo, Christos Mousas, Minsoo Choi & Klay Hauser

  2. University of California Santa Barbara, Santa Barbara, CA, 93106, USA

    Richard Mayer & Fangzheng Zhao

Authors
  1. Magzhan Mukanova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicoletta Adamo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christos Mousas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Minsoo Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Klay Hauser
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Richard Mayer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Fangzheng Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicoletta Adamo .

Editor information

Editors and Affiliations

  1. Aalborg University, Aalborg, Denmark

    Anthony L. Brooks

Rights and permissions

Reprints and permissions

Copyright information

© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mukanova, M. et al. (2024). Animated Pedagogical Agents Performing Affective Gestures Extracted from the GEMEP Dataset: Can People Recognize Their Emotions?. In: Brooks, A.L. (eds) ArtsIT, Interactivity and Game Creation. ArtsIT 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 565. Springer, Cham. https://doi.org/10.1007/978-3-031-55312-7_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-55312-7_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-55311-0

  • Online ISBN: 978-3-031-55312-7

Publish with us

Policies and ethics

Search

Navigation