Skip to main content
SpringerLink
Log in

The Effect of Multimodal Emotional Expression on Responses to a Digital Human during a Self-Disclosure Conversation: a Computational Analysis of User Language

  • Patient Facing Systems
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

Digital humans show promise for use in healthcare as virtual therapists to deliver psychotherapy or companions for social support. For digital humans to be effective and engaging in these roles, it is important they can build close relationships with people. Emotional expressiveness can improve social closeness in human relationships, especially for females. However, it is unknown whether multimodal emotional expression improves relationships with digital humans. Participants were 185 adults aged 18 years or older with English fluency. Participants were block-randomized by gender to complete the Relationship Closeness Induction Task with one of six versions of a digital human. The digital humans varied in modality richness (face, no face) and emotional expression (emotional voice, neutral voice; emotional face, neutral face). Participants’ language was analysed for emotional content using Linguistic Inquiry and Word Count software. A series of three-way ANOVA and ANCOVA were conducted to evaluate the effect of digital human face type, voice type, and participant gender on emotional content in participant language. A digital human with no face was associated with more first-person singular pronoun use than a neutral face and an emotional face digital human. A digital human with no face and a neutral voice received more general negative emotion language than a digital human with no face and an emotional voice. Findings suggest the presence of a face and emotion in the voice may improve emotional responses to digital humans. Results provide evidence for aspects of the theoretical framework of embodied agent-patient communication.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Sagar, M, Bullivant, D, Robertson, P, Efimov, O, Jawed, K, Kalarot, R, Wu, T. A neurobehavioural framework for autonomous animation of virtual human faces. In Proceedings of the SA’14 SIGGRAPH Asia Workshop on Autonomous Virtual Humans and Social Robot for Tele-presence. ACM. https://doi.org/10.1145/2668956.2668960 (2014)

  2. Cassell J, Sullivan J, Churchill E, Prevost S (2000). Embodied Conversational Agents. MIT Press, Cambridge.

    Book  Google Scholar 

  3. Sagar M, Seymour M, Henderson A (2016). Creating connection with autonomous facial animation. Communications of the ACM 59:82-91. https://doi.org/10.1145/2950041.

    Article  Google Scholar 

  4. Fuertes JN, Anand P, Haggerty G, Kestenbaum M, Rosenblum GC (2014). The physician-patient working alliance and patient psychological attachment, adherence, outcome expectation, and satisfaction in a sample of rheumatology patients. Behavioral Medicine 41:60-68. https://doi.org/10.1080/08964289.2013.875885.

    Article  PubMed  Google Scholar 

  5. Thompson L, McCabe R (2012). The effect of clinician-patient alliance and communication on treatment adherence in mental health care: a systematic review. BMC Psychiatry 12:87. https://doi.org/10.1186/1471-244X-12-87.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Alexander JA, Hearld LR, Mittler JN, Harvey J (2012). Patient-physician role relationships and patient activation among individuals with chronic illness. Health Services Research 47:1201-1223. https://doi.org/10.1111/j.1475-6733.2011.01354.x.

    Article  PubMed  Google Scholar 

  7. Stewart MA (1995). Effective physician-patient communication and health outcomes: a review. CMAJ: Canadian Medical Association Journal 152:1423.

    CAS  PubMed  Google Scholar 

  8. Kugbey N, Asante KO, Meyer-Weitz A (2018). Doctor-patient relationship mediates the effects of shared decision making on health-related quality of life among women living with breast cancer. South African Journal of Psychology 49:364-375. https://doi.org/10.1177/0081246318801159.

    Article  Google Scholar 

  9. Fricchione GL (2011). Compassion and Healing in Medicine and Society: On the Nature and Use of Attachment Solutions to Separation Challenges. Johns Hopkins University Press, Baltimore.

    Google Scholar 

  10. Lepore SJ (1998). Problems and prospects for the social support-reactivity hypothesis. Annals of Behavioral Medicine 20:257-269. https://doi.org/10.1007/BF02886375.

    Article  CAS  PubMed  Google Scholar 

  11. Padgett DA, Glaser R (2003). How stress influences the immune response. Trends in Immunology 25:444-448. https://doi.org/10.1016/s1471-4906(03)00173-x.

    Article  Google Scholar 

  12. Biondi M, Picardi A (1999). Psychological stress and neuroendocrine function in humans: the last two decades of research. Psychotherapy and Psychosomatics 68:114-150. https://doi.org/10.1159/000012323.

    Article  CAS  PubMed  Google Scholar 

  13. Kim H, Cheon E, Bai D, Lee YH, Koo B (2018). Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry Investigation 15:235-245. https://doi.org/10.30773/pi.2017.08.17.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Kiecolt-Glaser JK, Gouin J, Hantsoo L (2010). Close relationships, inflammation, and health. Neuroscience & Biobehavioral Reviews 35:33-38. https://doi.org/10.1016/j.neubiorev.2009.09.003.

    Article  Google Scholar 

  15. Floyd K, Mikkelson AC, Tafoya MA, Farinelli L, La Valley AG, Judd J, Haynes MT, Davis KL, Wilson J (2007). Human affection exchange: XIII. Affectionate communication accelerates neuroendocrine stress recovery. Health Communication 2:123-132. https://doi.org/10.1080/10410230701454015.

    Article  Google Scholar 

  16. Glynn LM, Christenfeld N, Gerin W (1999). Gender, social support, and cardiovascular responses to stress. Psychosomatic Medicine 61:234-242. https://doi.org/10.1097/00006842-199903000-00016.

    Article  CAS  PubMed  Google Scholar 

  17. House, J, Kahn, R. Measures and concepts of social support. In: Social Support and Health. Academic Press, New York. (1985)

  18. Rini C, Schetter CD, Hobel CJ, Glynn LM, Sandman CA (2006). Effective social support: antecedents and consequences of partner support during pregancy. Personal Relationships 13:207-229. https://doi.org/10.1111/j.1475-8611.2006.00114.x.

    Article  Google Scholar 

  19. Robinson H, MacDonald B, Kerse N, Broadbent E (2013). The psychosocial effects of a companion robot: a randomized controlled trial. Journal of the American Medical Directions Association 14:661-667. https://doi.org/10.1016/j.jamda.2013.02.007.

    Article  Google Scholar 

  20. Robinson H, MacDonald B, Broadbent E (2015). Physiological effects of a companion robot on blood pressure of older people in a residential care facility: a pilot study. Australasian Journal on Ageing 34:27-32. https://doi.org/10.1111/ajag.12099.

    Article  PubMed  Google Scholar 

  21. Albright G, Adam C, Serri D, Bleeker S, Goldman R (2016). Harnessing the power of conversations with virtual humans to change health behaviors. Mhealth 2:1-13. https://doi.org/10.21037/mhealth.2016.11.02.

    Article  Google Scholar 

  22. Richards D, Caldwell P (2017). Improving health outcomes sooner rather than later via an interactive website and virtual specialist. IEEE Journal of Biomedical and Health Informatics 22:1699-1706. https://doi.org/10.1109/JBHI.2017.2782210.

    Article  PubMed  Google Scholar 

  23. Bickmore, T.W., Puskar, K., Schlenk, E.A., Pfeifer, L.M., & Sereika, S.M. (2010). Maintaining reality: Relational agents for antipsychotic medication adherence. Interacting with Computers, 22(4), 276-288. https://doi.org/10.1016/j.intcom.2010.02.001.

    Article  Google Scholar 

  24. Philip, P, Dupuy, L., Auriacombe, M., Serre, F., de Sevin, E., Sauteraud, A., & Micoulaud-Franchi, J. (2020). Trust and acceptance of a virtual psychiatric interview between embodied conversational agents and outpatients. Npj Digital Medicine, 3(2), 1-7. https://doi.org/10.1038/s41746-019-0213-y.

    Article  Google Scholar 

  25. Loveys, K, Sebaratnam, G, Sagar, M, Broadbent, E. The effect of design features on relationship quality with embodied conversational agents: a systematic review. (In press)

  26. Parks MR, Floyd K (1996). Meanings for closeness and intimacy in friendship. Journal of Social and Personal Relationships 13:85-107. https://doi.org/10.1177/0265407596131005.

    Article  Google Scholar 

  27. Kraus MW (2017). Voice-only communication enhances empathic accuracy. American Psychologist 72:644-654. https://doi.org/10.1037/amp0000147.

    Article  PubMed  Google Scholar 

  28. Burleson BR (2003). The experience and effects of emotional support: what the study of cultural and gender differences can tell us about close relationships, emotion, and interpersonal communication. Personal Relationships 10:1-23. https://doi.org/10.1111/1475-6811.00033.

    Article  Google Scholar 

  29. Floyd K (1995). Gender and closeness among friends and siblings. The Journal of Psychology 129:193-202. https://doi.org/10.1080/00223980.1995.9914958.

    Article  CAS  PubMed  Google Scholar 

  30. Pelachaud C (2009). Modelling multimodal expression of emotion in a virtual agent. Philosophical Transactions: Biological Sciences 364:3539-3548. https://doi.org/10.1098/rstb.2009.0186.

    Article  Google Scholar 

  31. Brave S, Nass C, Hutchinson K (2005). Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. International Journal of Human-Computer Studies 62:161-178. https://doi.org/10.1016/j.ijhcs.2004.11.002.

    Article  Google Scholar 

  32. Maldonado, H, Lee J. E. R, Brave, S, Nass, C, Nakajima, H, Yamada, R, Iwamura, K, Morishima, Y. We learn better together: enhancing elearning with emotional characters. In Proceedings of the 2005 Conference on Computer Support for Collaborative Learning: Learning 2005: The Next 10 Years! 408-417. (2005)

  33. Pennebaker JW, Boyd RL, Jordan K, Blackburn K (2015). The development and psychometric properties of LIWC2015. University of Texas at Austin, Austin.

    Google Scholar 

  34. Tausczik YR, Pennebaker JW (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology 29:24-54. https://doi.org/10.1177/0261927X09351676.

    Article  Google Scholar 

  35. Rude S, Gortner EM, Pennebaker JW (2004). Language use of depressed and depression-vulnerable college students. Cognition & Emotion 18:1121-1133. https://doi.org/10.1080/02699930441000030.

    Article  Google Scholar 

  36. Kahn JH, Tobin RM, Massey AE, Anderson JA (2007). Measuring emotional expression with the Linguistic Inquiry and Word Count. The American Journal of Psychology 120:263-286. https://doi.org/10.2307/20445398.

    Article  PubMed  Google Scholar 

  37. Cohn MA, Mehl MR, Pennebaker JW (2004). Linguistic markers of psychological change surrounding September 11, 2001. Psychological Science 15:687-693. https://doi.org/10.1111/j.0956-7976.2004.00741.x.

    Article  PubMed  Google Scholar 

  38. Sedikides C, Campbell WK, Reeder GD, Elliot AJ (1999). The relationship closeness induction task. Representative Research in Social Psychology 23:1-4.

    Google Scholar 

  39. Lisetti C, Amini R, Yasavur U, Rishe N (2013). I can help you change! An empathic virtual agent delivers behavior change health interventions. ACM Transactions on Management Information Systems (TMIS), 4:1-28. https://doi.org/10.1145/2544103.

    Article  Google Scholar 

  40. Creed C, Beale R (2012). User interactions with an affective nutritional coach. Interacting with Computers 24:339-350. https://doi.org/10.1016/j.intcom.2012.05.004.

    Article  Google Scholar 

  41. Koda, T, Maes, P. Agents with faces: the effect of personification. In Proceedings of the 15th IEEE International Workshop on Robot and Human Communication 189-194. https://doi.org/10.1109/ROMAN.1996.568812.(1996)

  42. Burleson W, Picard R (2007). Gender-specific approaches to developing emotionally intelligent learning companions. IEEE Intelligent Systems 22:62-69. https://doi.org/10.1109/MIS.2007.69.

    Article  Google Scholar 

  43. Li HZ, Zhang Z, Yum Y, Lundgren J, Pahal JS (2008). Interruption and patient satisfaction in resident-patient consultations. Health Education 108:411-427. https://doi.org/10.1108/09654280810900026.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychological Medicine, The University of Auckland, Auckland, New Zealand

    Kate Loveys & Elizabeth Broadbent

  2. Soul Machines Ltd, Auckland, New Zealand

    Mark Sagar

  3. Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand

    Mark Sagar

Authors
  1. Kate Loveys
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Sagar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elizabeth Broadbent
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kate Loveys.

Ethics declarations

Conflict of Interest

MS is the CEO of Soul Machines (an artificial intelligence company), which supports KL with a PhD stipend, and contracts EB for consultancy work.

Ethics Approval

Approval was received by the University of Auckland Human Participants Ethics Committee on 01/11/2018 (reference no. 022191).

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article should be part of a Healthcare Intelligent Multi-Agent Systems collection.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Loveys, K., Sagar, M. & Broadbent, E. The Effect of Multimodal Emotional Expression on Responses to a Digital Human during a Self-Disclosure Conversation: a Computational Analysis of User Language. J Med Syst 44, 143 (2020). https://doi.org/10.1007/s10916-020-01624-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-020-01624-4

Keywords

Search

Navigation