Skip to main content
SpringerLink
Log in

Human perception of a conversational virtual human: an empirical study on the effect of emotion and culture

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Virtual reality applications with virtual humans, such as virtual reality exposure therapy, health coaches and negotiation simulators, are developed for different contexts and usually for users from different countries. The emphasis on a virtual human’s emotional expression depends on the application; some virtual reality applications need an emotional expression of the virtual human during the speaking phase, some during the listening phase and some during both speaking and listening phases. Although studies have investigated how humans perceive a virtual human’s emotion during each phase separately, few studies carried out a parallel comparison between the two phases. This study aims to fill this gap, and on top of that, includes an investigation of the cultural interpretation of the virtual human’s emotion, especially with respect to the emotion’s valence. The experiment was conducted with both Chinese and non-Chinese participants. These participants were asked to rate the valence of seven different emotional expressions (ranging from negative to neutral to positive during speaking and listening) of a Chinese virtual lady. The results showed that there was a high correlation in valence rating between both groups of participants, which indicated that the valence of the emotional expressions was as easily recognized by people from a different cultural background as the virtual human. In addition, participants tended to perceive the virtual human’s expressed valence as more intense in the speaking phase than in the listening phase. The additional vocal emotional expression in the speaking phase is put forward as a likely cause for this phenomenon.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Bailenson JN, Yee N, Merget D, Schroeder R (2006) The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction. Presence Teleoper Virtual Environ 15(4):359–372

    Google Scholar 

  • Barrett LF (2006) Solving the emotion paradox: categorization and the experience of emotion. Pers Soc Psychol Rev Off J Soc Pers Soc Psychol 10:20–46. doi:10.1207/s15327957pspr1001_2

    Article  Google Scholar 

  • Bartneck C (2001) Affective expressions of machines. CHI ‘01 extended abstracts on human factors in computing systems, pp 189–190

  • Bates J (1994) The role of emotion in believable agents. Commun ACM 37(7):122–125

    Article  Google Scholar 

  • Bradley MM, Lang PJ (2007) The international affective digitized sounds (2nd Edition; IADS-2): affective ratings of sounds and instruction manual. Technical report B-3. University of Florida, Gainesville, Fl

  • Brand M (1999) Voice puppetry. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques SIGGRAPH 99, pp 21–28

  • Bregler C (1997) Video rewrite: driving visual speech with audio. In: Proceedings of SIGGRAPH’97, pp 1–8

  • Brinkman W-P, Veling W, Dorrestijn E, Sandino G, Vakili V, van der Gaag M (2011) Virtual reality to study responses to social environmental stressors in individuals with and without psychosis. Stud Health Technol Inf 167:86–91

    Google Scholar 

  • Brinkman W-P, Hartanto D, Kang N, de Vliegher D, Kampmann IL, Morina N et al (2012) A virtual reality dialogue system for the treatment of social phobia. In: Paper presented at the CHI’12 extended abstracts on human factors in computing systems

  • Broekens J, Brinkman W-P (2009) Affectbutton: towards a standard for dynamic affective user feedback. Affect Comput Intel Interact Workshops 2009:1–8

    Google Scholar 

  • Broekens J, Brinkman W-P (2013) AffectButton: a method for reliable and valid affective self-report. Int J Hum Comput Stud 71(6):641–667

    Article  Google Scholar 

  • Broekens J, Pronker A, Neuteboom M (2010) Real time labeling of affect in music using the affectbutton. In: Proceedings of the 3rd international workshop on affective interaction in natural environments, pp 21–26

  • Broekens J, Harbers M, Brinkman W-P, Jonker C, Van den Bosch K, Meyer JJ (2011) Validity of a virtual negotiation training. In: IVA’11 Proceedings of the 11th international conference on intelligent virtual agents, pp 435–436

  • Broekens J, Harbers M, Brinkman W-P, Jonker C, Van den Bosch K, Meyer J-J (2012) Virtual reality negotiation training increases negotiation knowledge and skill. In: IVA’12 Proceedings of the 12th international conference on intelligent virtual agents, pp 218–230

  • Broekens J, Qu C, Brinkman W-P (2012) Dynamic facial expression of emotion made easy. Technical report. Interactive Intelligence, Delft University of Technology, pp 1–30

  • Cassell J, Thorisson KR (1999) The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents. Appl Artif Intell 13:519–538

    Article  Google Scholar 

  • Cassell J, Pelachaud C, Badler N, Steedman M, Achorn B, Becket T et al (1994) Animated conversation: Rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In: Proceedings of ACM SIGGRAPH, pp 413–420

  • Cerezo E, Baldassarri S (2008) Affective embodied conversational agents for natural interaction. In: Or J (ed) Affective computing: emotion modelling, synthesis and recognition, pp 329–354

  • Chuang E, Bregler C (2002) Performance driven facial animation using blendshape interpolation. Computer Science Technical Report, Stanford University

  • Church T, Katigbak M (1998) Language and organisation of Filipino emotion concepts: comparing emotion concepts and dimensions across cultures. Cogn Emot 12(1):63–92

    Article  Google Scholar 

  • Cohen MM, Massaro DW (1993) Modeling coarticulation in synthetic visual speech. In: Thalman NM, Thalman D (eds) Models and Techniques in Computer Animation. Springer, Verlag, pp 139–156

  • Core M, Traum D, Lane HC, Swartout W, Marsella S, Gratch J, Van Lent M (2006) Teaching negotiation skills through practice and reflection with virtual humans. Simulation 82:685–701

    Google Scholar 

  • Cowell AJ, Stanney KM (2003) Embodiment and interaction guidelines for designing credible, trustworthy embodied conversational agents. In: 4th international workshop on intelligent virtual agents IVA 2003, 2792, pp 301–309

  • Darwin C (1872) The expression of emotion in man and animals. Philosophical Library, New York

    Book  Google Scholar 

  • Dotsch R, Wigboldus DHJ (2008) Virtual prejudice. J Exp Soc Psychol 44(4):1194–1198

    Article  Google Scholar 

  • Ekman P (1994) Strong evidence for universals in facial expressions: a reply to Russell’s mistaken critique. Psychological Bull 115(2):268–287

    Article  Google Scholar 

  • Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc 17(2):124–129

    Article  Google Scholar 

  • Ekman P, Friesen WV, O’Sullivan M, Chan A, Diacoyanni-Tarlatzis I, Heider K, Ricci-Bitti P (1987) Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol 53(4):712–717

    Article  Google Scholar 

  • Ekman P, Rolls ET, Perrett DI, Ellis HD (1992) Facial expressions of emotion: an old controversy and new findings. Philos Trans Biol Sci 335:63–69

    Article  Google Scholar 

  • Ekman P, Friesen WV, Hager JC (2002) Facial action coding system. Human Face 97:4–5

    Google Scholar 

  • Elfenbein H (2003) Universals and cultural differences in recognizing emotions. Curr Dir Psychol Sci 12(5):159–164

    Google Scholar 

  • Elfenbein HA, Ambady N (2002) Is there an in-group advantage in emotion recognition? Psychol Bull 128(2):243–249

    Article  Google Scholar 

  • Elfenbein HA, Beaupre M, Levesque M, Hess U (2007) Toward a dialect theory: cultural differences in the expression and recognition of posed facial expressions. Emotion (Washington, DC) 7(1):131–146

  • Endrass B, Rehm M, Lipi A (2011) Culture-related differences in aspects of behavior for virtual characters across Germany and Japan. In: Proceedings of AAMAS’11, 2, pp 441–448

  • Ersotelos N, Dong F (2008) Building highly realistic facial modeling and animation: a survey. Visual Comput 24(1):13–30

    Article  Google Scholar 

  • Ezzat T, Geiger G, Poggio T (2004) Trainable videorealistic speech animation. In: Sixth IEEE international conference on automatic face and gesture recognition 2004 proceedings, pp 57–64

  • Fox E (2008) Emotion science cognitive and neuroscientific approaches to understanding human emotions. Palgrave Macmillan, UK

    Google Scholar 

  • Gratch J, Rickel J, Andre E, Cassell J, Petajan E, Badler NI, Jeff R (2002) Creating interactive virtual humans: some assembly required. IEEE Intell Syst 17(4):54–63

    Article  Google Scholar 

  • Gross MM, Crane EA, Fredrickson BL (2010) Methodology for assessing bodily expression of emotion. J Nonverbal Behav 34:223–248. doi:10.1007/s10919-010-0094-x

    Article  Google Scholar 

  • Haring M, Bee N, Andre E (2011) Creation and evaluation of emotion expression with body movement, sound and eye color for humanoid robots. In: RO-MAN, 2011 IEEE, pp 204–209

  • Hofstede G (2001) Culture’s consequences: comparing values, behaviors, institutions and organisations across nations. Sage Publications, Thousand Oaks

    Google Scholar 

  • Hudlicka E, Delft TU (2009) Foundations for modelling emotions in game characters: modelling emotion effects on cognition. In: Affective computing and intelligent interaction and workshops, ACII 2009

  • Irtel H (2007) PXLab: the psychological experiments laboratory [online]. University of Mannheim, Mannheim

    Google Scholar 

  • Isbister K (2006) Better game characters by design: a psychological approach. Education, CRC Press

  • Jack RE, Garrod OGB, Yu H, Caldara R, Schyns PG (2012) Facial expressions of emotion are not culturally universal. Proc Natl Acad Sci 109(19):7241–7244

    Article  Google Scholar 

  • Jan D, Herrera D, Martinovski B (2007) A computational model of culture-specific conversational behavior. In: IVA ‘07 Proceedings of the 7th international conference on intelligent virtual agents, pp 45–56

  • Kahler K, Haber J, Seidel H-P (2001) Geometry-based muscle modeling for facial animation. In: Proceedings of graphics interface, pp 37–46

  • Keltner D, Ekman P (2000) Facial expression of emotion. Handbook of emotions, 2nd edn. pp 236–249

  • Kleinsmith A, Bianchi-Berthouze N (2013) Affective body expression perception and recognition: a survey. IEEE Trans Affect Comput 4:15–33. doi:10.1109/T-AFFC.2012.16

    Article  Google Scholar 

  • Kleinsmith A, De Silva PR, Bianchi-Berthouze N (2006) Cross-cultural differences in recognizing affect from body posture. Interact Comput 18(6):1371–1389

    Article  Google Scholar 

  • Kulms P, Kramer NC, Gratch J, Kang S-H (2011) It’s in their eyes: a study on female and male virtual humans' gaze. In: IVA’11 Proceedings of the 11th international conference on intelligent virtual agents, pp 80–92

  • Lance BJ, Rey MD, Marsella SC (2008) A model of gaze for the purpose of emotional expression in virtual embodied agents. In: AAMAS ‘08 proceedings of the 7th international joint conference on autonomous agents and multiagent systems, 1, pp 12–16

  • Lang PJ (1995) The emotion probe. Studies of motivation and attention. Am Psychol 50(5):372–385

    Article  Google Scholar 

  • Lang PJ, Bradley MM, Cuthbert BN (1999) International affective picture system (IAPS): technical manual and affective ratings. Psychology. The Center for Research in Psychophysiology, University of Florida, Gainesville, FL

  • Lang PJ, Bradley MM, Cuthbert BN (2008) International affective picture system (IAPS): affective ratings of pictures and instruction manual. Technical Report A-8

  • Lee J, Marsella SC (2012) Modeling speaker behavior: a comparison of two approaches. In: IVA’12 Proceedings of the 12th international conference on intelligent virtual agents, pp 161–174

  • Lee Y, Terzopoulos D, Walters K (1995) Realistic modeling for facial animation. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques SIGGRAPH 95, pp 55–62

  • Lee J, Prendinger H, Neviarouskaya A, Marsella S (2009) Learning models of speaker head nods with affective information. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops, pp 1–6

  • Link M, Armsby P, Hubal RC, Guinn CI (2006) Accessibility and acceptance of responsive virtual human technology as a survey interviewer training tool. Comput Hum Behav 22:412–426. doi:10.1016/j.chb.2004.09.008

    Article  Google Scholar 

  • Litwinowicz P, Williams L (1994) Animating images with drawings. In: Proceedings of the 21st annual conference on computer graphics and interactive techniques SIGGRAPH 94, pp 409–412

  • MacDorman KFK, Coram JJA, Ho C-CC, Patel H (2010) Gender differences in the impact of presentational factors in human character animation on decisions in ethical dilemmas. Presence Teleoper Virtual Environ 19(3):213–229

    Google Scholar 

  • Matsumoto D (2002) Methodological requirements to test a possible in-group advantage in judging emotions across cultures: comment on Elfenbein and Ambady (2002) and evidence. Psychol Bull 128(2):236–242

    Article  Google Scholar 

  • Matsumoto D (2007) Emotion judgments do not differ as a function of perceived nationality. Int J Psychol 42(3):207–214

    Article  Google Scholar 

  • Mehrabian A, Russell JA (1974) An approach to environmental psychology. MIT Press, Cambridge, MA

    Google Scholar 

  • Melo Cd, Carnevale P, Gratch J (2011) The effect of expression of anger and happiness in computer agents on negotiations with humans. In: The tenth international conference on autonomous agents and multiagent systems, pp 2–6

  • Moridis CN, Economides AA (2012) Affective learning: empathetic agents with emotional facial and tone of voice expressions. IEEE Trans Affect Comput 3(3):260–272

    Article  Google Scholar 

  • Morris JD (1995) Observations: SAM the self-assessment manikin an efficient cross-cultural measurement of emotional response. J Advert Res 35(6):63–68

    Google Scholar 

  • Ochsman RB, Chapanis A (1974) The effects of 10 communication modes on the behavior of teams during co-operative problem-solving. Int J ManMachine Stud 6(5):579–619

    Article  Google Scholar 

  • Opris D, Pintea S, Garcia-Palacios A, Botella CM, Szamoskozi S, David D (2012) Virtual reality exposure therapy in anxiety disorders: a quantitative meta-analysis. Depression Anxiety 29:85–93. doi:10.1002/da.20910

    Article  Google Scholar 

  • Parke FI (1972) Computer generated animation of faces. Proc ACM Annu Conf 1:451–457

    Google Scholar 

  • Parke FI (1974) A parametric model for human faces. The University of Utah, Doctoral Dissertation

  • Petrushin V (1999) Emotion in speech: recognition and application to call centers. In: Artificial neural network in engineering (ANNIE’99), pp 7–10

  • Picard RW (1998) Toward agents that recognize emotion. In: Actes proceedings IMAGINA, pp 153–165

  • Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191

    Article  Google Scholar 

  • Platt SM, Badler NI (1981) Animating facial expressions. ACM SIGGRAPH Comput Graph 15(3):245–252

    Article  Google Scholar 

  • Qiu L, Benbasat I (2005) Online consumer trust and live help interfaces: the effects of text-to-speech voice and three-dimensional avatars. Int J Human-Comput Interact 19:37–41

    MATH  Google Scholar 

  • Reeves B, Nass C (1996) The media equation. Cambridge University Press, Cambridge

    Google Scholar 

  • Rizzo A, Lange B, Buckwalter J, Forbell E, Kim J, Sagae K, Kenny P (2011) An intelligent virtual human system for providing healthcare information and support. Stud Health Technol Inf 163:503–509

    Google Scholar 

  • Russell JA (1991) Culture and the categorization of emotions. Psychol Bull 110:426–450

    Article  Google Scholar 

  • Ruttkay Z, Pelachaud C (2005) From brows to trust: evaluating embodied conversational agents. Springer, Berlin

  • Scherer KR (1995) Expression of emotion in voice and music. J Voice Off J Voice Found 9(3):235–248

    Article  Google Scholar 

  • Scherer KR (2003) Vocal communication of emotion: a review of research paradigms. Speech Commun 40:227–256

    Article  MATH  Google Scholar 

  • Schiano DJ, Ehrlich SM, Rahardja K, Sheridan K (2000) Face to interface: facial affect in (hu)man and machine. Proc ACM CHI 2000:193–200

    Google Scholar 

  • Schroder M (2004) Speech and emotion research: an overview of research frameworks and a dimensional approach to emotional speech synthesis. Research Report of the Institute of Phonetics

  • Slater M, Pertaub D-P, Steed A (1999) Public speaking in virtual reality: facing an audience of avatars. IEEE Comput Graphics Appl 19(2):6–9

    Article  Google Scholar 

  • Tomkins SS (1962) Affect, imagery, consciousness: vol 1. The positive affects. Springer, New York

    Google Scholar 

  • Tomkins SS (1963) Affect, imagery, consciousness: vol 2. The negative affects. Springer, New York

    Google Scholar 

  • Tsapatsoulis N, Raouzaiou A, Kollias S, Cowie R, Douglas-Cowie E (2002) Emotion recognition and synthesis based on MPEG-4 FAPs. In; MPEG-4 facial animation the standard implementations applications

  • Waters K (1987) A muscle model for animating three-dimensional facial expression. Comput Graph SIGGRAPH Proc 21(4):17–24

    Article  MathSciNet  Google Scholar 

  • Wierzbicka A (1995) Emotions across languages and cultures: diversity and universals. Cambridge University Press, Cambridge

    Google Scholar 

  • Wong JW-E, McGee K (2012) Frown more, talk more: effects of facial expressions in establishing conversational rapport with virtual agents. In: IVA’12 Proceedings of the 12th international conference on intelligent virtual agents, pp 419–425

  • Yun C, Deng Z, Hiscock M (2009) Can local avatars satisfy a global audience? A case study of high-fidelity 3D facial avatar animation in subject identification and emotion perception by US and international groups. Comput Entertain 7(2):1–25

    Article  Google Scholar 

  • Zhang QZQ, Liu Z, Quo GQG, Terzopoulos D, Shum H-YSH-Y (2006) Geometry-driven photorealistic facial expression synthesis. IEEE Trans Visual Comput Graphics 12(1):48–60

    Article  Google Scholar 

Download references

Acknowledgments

This study is supported in part by the Chinese Scholarship Council (No. 2008609199) and the COMMIT project—Interaction for Universal Access.

Author information

Authors and Affiliations

  1. Delft University of Technology, Mekelweg 4, 2628 CD, Delft, The Netherlands

    Chao Qu, Willem-Paul Brinkman, Yun Ling, Pascal Wiggers & Ingrid Heynderickx

  2. Philips Research Laboratories, High Tech Campus 34, 5656 AE, Eindhoven, The Netherlands

    Ingrid Heynderickx

Authors
  1. Chao Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Willem-Paul Brinkman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yun Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pascal Wiggers
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ingrid Heynderickx
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chao Qu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Qu, C., Brinkman, WP., Ling, Y. et al. Human perception of a conversational virtual human: an empirical study on the effect of emotion and culture. Virtual Reality 17, 307–321 (2013). https://doi.org/10.1007/s10055-013-0231-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-013-0231-z

Keywords

Search

Navigation