Skip to main content
SpringerLink
Log in

The effect of avatar facial expressions on trust building in social virtual reality

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In immersive virtual reality (VR) applications, the facial expressions and lip-syncing of an avatar can have a significant impact on a user’s experience. In this paper, we designed a VR “trust game” scene to evaluate the effects of four expression conditions (positive facial expressions, neutral facial expressions, negative facial expressions and no expressions and lip-syncing) on participants in an immersive VR scene. We measured the participants with both objective and subjective measures. The two objective behavioral measures were the level of investment in the “trust game” and the users’ eye-movement data, and the subjective measures included social presence, emotional awareness level, and user preferences. We found that the participants were generally less trusting of the avatars with negative expressions, while the avatars with positive expressions made the participants feel comfortable and thus increased their willingness to cooperate with the avatars. In conclusion, avatars with facial expressions, whether positive or negative, were more effective in influencing the participants’ trust levels and decision-making behaviors than those without facial expressions. These findings provide novel ideas and suggestions for improving the level of human–computer interaction in VR and enhancing user experience in VR scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Fernández-Dols, J.M., Russell, J.A.: The Psychology of Facial Expression. Cambridge University Press, Cambridge (1997). https://doi.org/10.1017/CBO9780511659911

    Book  Google Scholar 

  2. Izard, C.E.: Emotions as motivations: an evolutionary-developmental perspective. In: Nebraska Symposium on Motivation, vol. 26, pp. 163–200 (1978). https://psycnet.apa.org/record/1982-11912-001

  3. Ekman, P., Oster, H.: Facial expressions of emotion. Annu. Rev. Psychol. 30(1), 527–554 (1979). https://doi.org/10.1146/annurev.ps.30.020179.002523

    Article  Google Scholar 

  4. Krach, S., Hegel, F., Wrede, B., Sagerer, G., Binkofski, F., Kircher, T.: Can machines think? Interaction and perspective taking with robots investigated via FMRI. PLoS ONE 3(7), 1–11 (2008). https://doi.org/10.1371/journal.pone.0002597

    Article  Google Scholar 

  5. Berg, J., Dickhaut, J., McCabe, K.: Trust, reciprocity, and social history. Games Econom. Behav. 10(1), 122–142 (1995). https://doi.org/10.1006/game.1995.1027

    Article  MATH  Google Scholar 

  6. Scharlemann, J.P., Eckel, C.C., Kacelnik, A., Wilson, R.K.: The value of a smile: game theory with a human face. J. Econ. Psychol. 22(5), 617–640 (2001). https://doi.org/10.1016/S0167-4870(01)00059-9

    Article  Google Scholar 

  7. Krumhuber, E., Manstead, A.S., Cosker, D., Marshall, D., Rosin, P.L., Kappas, A.: Facial dynamics as indicators of trustworthiness and cooperative behavior. Emotion 7(4), 730 (2007). https://doi.org/10.1037/1528-3542.7.4.730

    Article  Google Scholar 

  8. Agarwal, S., Santra, B., Mukherjee, D.P.: Anubhav: recognizing emotions through facial expression. Vis. Comput. 34(2), 177–191 (2018). https://doi.org/10.1007/s00371-016-1323-z

    Article  Google Scholar 

  9. Smith, H.J., Neff, M.: Communication behavior in embodied virtual reality. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2018). https://doi.org/10.1145/3173574.3173863

  10. Pan, Y., Steed, A.: The impact of self-avatars on trust and collaboration in shared virtual environments. PLoS ONE 12(12), 0189078 (2017). https://doi.org/10.1371/journal.pone.0189078

    Article  Google Scholar 

  11. Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., Sasse, M.A.: The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 529–536 (2003). https://doi.org/10.1145/642611.642703

  12. Heidicker, P., Langbehn, E., Steinicke, F.: Influence of avatar appearance on presence in social VR. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI), pp. 233–234 (2017). https://doi.org/10.1109/3DUI.2017.7893357. IEEE

  13. Yoon, B., Kim, H.-I., Lee, G.A., Billinghurst, M., Woo, W.: The effect of avatar appearance on social presence in an augmented reality remote collaboration. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 547–556. IEEE (2019). https://doi.org/10.1109/VR.2019.8797719

  14. Cho, S., Kim, S.-W., Lee, J., Ahn, J., Han, J.: Effects of volumetric capture avatars on social presence in immersive virtual environments. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 26–34. IEEE (2020). https://doi.org/10.1109/VR46266.2020.00020

  15. Aseeri, S., Interrante, V.: The influence of avatar representation on interpersonal communication in virtual social environments. IEEE Trans. Vis. Comput. Graph. 27(5), 2608–2617 (2021). https://doi.org/10.1109/TVCG.2021.3067783

    Article  Google Scholar 

  16. Latoschik, M.E., Roth, D., Gall, D., Achenbach, J., Waltemate, T., Botsch, M.: The effect of avatar realism in immersive social virtual realities. In: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, pp. 1–10 (2017). https://doi.org/10.1145/3139131.3139156

  17. Zibrek, K., Kokkinara, E., McDonnell, R.: The effect of realistic appearance of virtual characters in immersive environments-does the character’s personality play a role? IEEE Trans. Vis. Comput. Graph. 24(4), 1681–1690 (2018). https://doi.org/10.1109/TVCG.2018.2794638

    Article  Google Scholar 

  18. Gonzalez-Franco, M., Steed, A., Hoogendyk, S., Ofek, E.: Using facial animation to increase the enfacement illusion and avatar self-identification. IEEE Trans. Vis. Comput. Graph. 26(5), 2023–2029 (2020). https://doi.org/10.1109/TVCG.2020.2973075

    Article  Google Scholar 

  19. Hart, J.D., Piumsomboon, T., Lawrence, L., Lee, G.A., Smith, R.T., Billinghurst, M.: Emotion sharing and augmentation in cooperative virtual reality games. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts, pp. 453–460 (2018). https://doi.org/10.1145/3270316.3271543

  20. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., Debevec, P.: Multiview face capture using polarized spherical gradient illumination. In: Proceedings of the 2011 SIGGRAPH Asia Conference, pp. 1–10 (2011). https://doi.org/10.1145/2024156.2024163

  21. Kampouris, C., Zafeiriou, S., Ghosh, A.: Diffuse-specular separation using binary spherical gradient illumination. In: Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations. SR’18, pp. 1–10. Eurographics Association, Goslar, DEU (2018). https://doi.org/10.2312/sre.20181167

  22. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. SIGGRAPH’00, pp. 145–156. ACM Press/Addison-Wesley Publishing Co., USA (2000). https://doi.org/10.1145/344779.344855

  23. Debevec, P., Wenger, A., Tchou, C., Gardner, A., Waese, J., Hawkins, T.: A lighting reproduction approach to live-action compositing. ACM Trans. Graph. 21(3), 547–556 (2002). https://doi.org/10.1145/566654.566614

    Article  Google Scholar 

  24. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., Debevec, P.: Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans. Graph. 24(3), 756–764 (2005). https://doi.org/10.1145/1073204.1073258

    Article  Google Scholar 

  25. Alexander, O., Rogers, M., Lambeth, W., Chiang, J.-Y., Ma, W.-C., Wang, C.-C., Debevec, P.: The digital Emily project: achieving a photorealistic digital actor. IEEE Comput. Graph. Appl. 30(4), 20–31 (2010). https://doi.org/10.1109/MCG.2010.65

    Article  Google Scholar 

  26. Metahuman Creator. https://www.unrealengine.com/en-US/metahuman-creator/ (2021)

  27. Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A.G., Hodgins, J., Matthews, I.: A deep learning approach for generalized speech animation. ACM Trans. Graph. (TOG) 36(4), 1–11 (2017). https://doi.org/10.1145/3072959.3073699

    Article  Google Scholar 

  28. Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017). https://doi.org/10.1145/3072959.3073658

    Article  Google Scholar 

  29. Olszewski, K., Lim, J.J., Saito, S., Li, H.: High-fidelity facial and speech animation for VR HMDS. ACM Trans. Graph. (TOG) 35(6), 1–14 (2016). https://doi.org/10.1145/2980179.2980252

    Article  Google Scholar 

  30. Burgos-Artizzu, X.P., Fleureau, J., Dumas, O., Tapie, T., LeClerc, F., Mollet, N.: Real-time expression-sensitive HMD face reconstruction. In: SIGGRAPH Asia 2015 Technical Briefs, pp. 1–4 (2015). https://doi.org/10.1145/2820903.2820910

  31. Guo, K., Xu, F., Yu, T., Liu, X., Dai, Q., Liu, Y.: Real-time geometry, albedo, and motion reconstruction using a single RGB-D camera. ACM Trans. Graph. (ToG) 36(4), 1 (2017). https://doi.org/10.1145/3072959.3083722

    Article  Google Scholar 

  32. Wei, S.-E., Saragih, J., Simon, T., Harley, A.W., Lombardi, S., Perdoch, M., Hypes, A., Wang, D., Badino, H., Sheikh, Y.: VR facial animation via multiview image translation. ACM Trans. Graph. (TOG) 38(4), 1–16 (2019). https://doi.org/10.1145/3306346.3323030

    Article  Google Scholar 

  33. Suzuki, K., Nakamura, F., Otsuka, J., Masai, K., Itoh, Y., Sugiura, Y., Sugimoto, M.: Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display. In: 2017 IEEE Virtual Reality (VR), pp. 177–185. IEEE (2017). https://doi.org/10.1109/VR.2017.7892245

  34. Tortosa, M.I., Strizhko, T., Capizzi, M., Ruz, M.: Interpersonal effects of emotion in a multi-round trust game. Psicologica Int. J. Methodol. Exp. Psychol. 34(2), 179–198 (2013). https://eric.ed.gov/?id=EJ1018812

  35. Stouten, J., De Cremer, D.: Seeing is believing: the effects of facial expressions of emotion and verbal communication in social dilemmas. J. Behav. Decis. Mak. 23(3), 271–287 (2010). https://doi.org/10.1002/bdm.659

    Article  Google Scholar 

  36. Jaeger, B., Evans, A.M., Stel, M., van Beest, I.: Explaining the persistent influence of facial cues in social decision-making. J. Exp. Psychol. Gen. 148(6), 1008 (2019). https://doi.org/10.1037/xge0000591

    Article  Google Scholar 

  37. Mussel, P., Göritz, A.S., Hewig, J.: The value of a smile: facial expression affects ultimatum-game responses. Judgm. Decis. Mak. 8(3) (2013). https://nbn-resolving.org/urn:nbn:de:bvb:20-opus-129639

  38. Mathur, M.B., Reichling, D.B.: An uncanny game of trust: social trustworthiness of robots inferred from subtle anthropomorphic facial cues. In: 2009 4th ACM/IEEE International Conference on Human–Robot Interaction (HRI), pp. 313–314 (2009). https://doi.org/10.1145/1514095.1514192

  39. Wu, J., Paeng, E., Linder, K., Valdesolo, P., Boerkoel, J.: Trust and cooperation in human–robot decision making. https://www.aaai.org/ocs/index.php/FSS/FSS16/paper/view/14118/13675 (2016)

  40. Paradeda, R.B., Hashemian, M., Rodrigues, R.A., Paiva, A.: How facial expressions and small talk may influence trust in a robot. In: Agah, A., Cabibihan, J.-J., Howard, A.M., Salichs, M.A., He, H. (eds.) Social Robotics, pp. 169–178. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47437-3_17

    Chapter  Google Scholar 

  41. George, C., Eiband, M., Hufnagel, M., Hussmann, H.: Trusting strangers in immersive virtual reality. In: Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion. IUI’18 Companion. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3180308.3180355

  42. Kugler, T., Ye, B., Motro, D., Noussair, C.N.: On trust and disgust: evidence from face reading and virtual reality. Soc. Psychol. Pers. Sci. 11(3), 317–325 (2020). https://doi.org/10.1177/1948550619856302

    Article  Google Scholar 

  43. Harjunen, V., Spape, M., Ahmed, I., Jacucci, G., Ravaja, N.: Persuaded by the machine: the effect of virtual nonverbal cues and individual differences on compliance in economic bargaining. Comput. Hum. Behav. 87, 11 (2018). https://doi.org/10.1016/j.chb.2018.06.012

    Article  Google Scholar 

  44. Mori, M., MacDorman, K.F., Kageki, N.: The uncanny valley [from the field]. IEEE Robotics Autom. Mag. 19(2), 98–100 (2012). https://doi.org/10.1109/MRA.2012.2192811

    Article  Google Scholar 

  45. Ekman, P., Friesen, W.V.: Facial Action Coding System. American Psychological Association (APA) (1978). https://doi.org/10.1037/t27734-000

  46. Meneses, J.A.C., Díaz, J.M.M.: Vocal emotion expressions effects on cooperation behavior. Psicológica 38(1), 1–24 (2017). https://www.redalyc.org/pdf/169/16949050001.pdf

  47. Games, E.: Recording Facial Animation from an iOS Device. https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/FacialRecordingiPhone/ (2021)

  48. Harms, P.C., Biocca, P.F.: Internal Consistency and Reliability of the Networked Minds Measure of Social Presence. http://cogprints.org/7026/ (2004)

  49. Biocca, P.F., Harms, P.C.: Guide to the Networked Minds Social Presence Inventory v. 1.2. http://cogprints.org/6743/ (2003)

  50. Boyle, G.J.: Reliability and validity of Izard’s differential emotions scale. Pers. Individ. Differ. 7, 747–750 (1984). https://doi.org/10.1016/0191-8869(84)90124-7

    Article  Google Scholar 

  51. Tu, Z., Weng, D., Cheng, D., Shen, R., Fang, H., Bao, Y.: The effect of two different types of human–computer interactions on user’s emotion in virtual counseling environment. In: 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 217–221 (2019). https://doi.org/10.1109/ISMAR-Adjunct.2019.00-43

  52. Pauw, L.S., Sauter, D.A., van Kleef, G.A., Lucas, G.M., Gratch, J., Fischer, A.H.: The avatar will see you now: support from a virtual human provides socio-emotional benefits. Comput. Hum. Behav. 136, 107368 (2022). https://doi.org/10.1016/j.chb.2022.107368

  53. Shao, D., Lee, I.-J.: Acceptance and influencing factors of social virtual reality in the urban elderly. Sustainability 12(22) (2020). https://doi.org/10.3390/su12229345

  54. Baker, S., Kelly, R.M., Waycott, J., Carrasco, R., Hoang, T., Batchelor, F., Ozanne, E., Dow, B., Warburton, J., Vetere, F.: Interrogating social virtual reality as a communication medium for older adults. Proc. ACM Hum. Comput. Interact. 3(CSCW) (2019). https://doi.org/10.1145/3359251

Download references

Funding

This work was supported by the Key-Area Research and Development Program of Guangdong Province (No. 2019B010149001) and the National Natural Science Foundation of China (No. 62072036) and the 111 Project (B18005).

Author information

Authors and Affiliations

  1. Beijing Engineering Research Center of Mixed Reality and Advanced Display, Beijing Institute of Technology, Beijing, 100081, China

    Le Luo, Dongdong Weng, Jie Hao & Ziqi Tu

  2. School of Arts & Communication, Beijing Normal University, Beijing, 100875, China

    Ni Ding

Authors
  1. Le Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dongdong Weng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ni Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ziqi Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Dongdong Weng or Ni Ding.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 2414 KB)

Supplementary file 2 (mp4 5434 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, L., Weng, D., Ding, N. et al. The effect of avatar facial expressions on trust building in social virtual reality. Vis Comput 39, 5869–5882 (2023). https://doi.org/10.1007/s00371-022-02700-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02700-1

Keywords

Search

Navigation